The ODBC Driver Scavenger Hunt

At Columnar, we build ADBC (Arrow Database Connectivity) drivers. To ensure they perform well, we benchmark them against the mainstream alternatives, which often include ODBC drivers. Those benchmarks measure the advantage most people associate with ADBC: speed. Since many analytical databases produce result sets most efficiently in Arrow format, and since many downstream analytical tools consume data most efficiently in Arrow format, keeping the data in Arrow in between avoids format conversions and can have massive performance advantages. But we don’t assume that this principle alone makes an ADBC driver faster. We run loads of comparative benchmarks as part of our development process to characterize performance under many different conditions, as we’ve described in our deep dive into ADBC driver optimization.

All that benchmarking means we use ODBC a lot. We install and configure ODBC drivers alongside the ADBC drivers we’re testing, so we spend plenty of time finding vendor downloads, locating shared libraries, and getting driver managers to load them. And getting those ODBC drivers ready to use is a miserable experience.

But before we get into the misery of installing and configuring ODBC drivers, we should talk about what ODBC got right. When the Columnar founders designed ADBC, they learned from ODBC’s shortcomings and borrowed many of its best ideas.

What ODBC got right

ODBC defines a standardized API for connecting to databases, submitting queries, and reading results. For each database, a driver implements that API using any wire protocol the database supports; ODBC does not prescribe a particular one. Applications can code to the common API and reuse much of their database access code across drivers and databases. That portability has helped ODBC endure as a dominant database connectivity standard for more than thirty years. ADBC provides that same portability, with its own standard API implemented by database-specific drivers, and likewise leaves the choice of wire protocol to the driver.

ODBC drivers are distributed as shared libraries: .so files on Linux, .dylib files on macOS, and .dll files on Windows. The same compiled driver can then be used from C, C++, C# and other .NET languages, Go, Java and other JVM languages, Python, R, Ruby, Rust, and more, with the appropriate client library or bridge. ADBC uses the same model: in both standards, your choice of connector doesn’t tie you to one language, and the driver author doesn’t have to maintain an implementation for each one.

For an application to use that shared library, something has to load it. In both ODBC and ADBC, that job belongs to the driver manager. The application calls the driver manager, which finds and loads the requested driver at runtime and forwards calls to it.

Where ODBC left room for improvement

For any of this to work, though, the driver manager has to be able to find and load the driver. With ODBC, reaching that point means following installation instructions that depend on the driver vendor, your operating system, and your client library. So before you can benefit from one consistent API, you have to muddle through wildly inconsistent installation procedures.

Choose a client library

First, you have to choose a client library. Python alone has several ODBC options, including pyodbc, turbodbc, arrow-odbc, and pypyodbc. Turbodbc can be much faster than pyodbc for bulk transfers, but it can also crash with some ODBC drivers. Compare this to ADBC: the Apache Arrow project maintains one official client library for each supported language, giving you a clear default. In Python, that’s adbc_driver_manager.

Install a compatible driver manager

In ODBC, the driver manager is typically a separate system-level library, supplied by the operating system or installed separately. Installing or updating it can require administrator privileges and can break other applications that share it. ADBC was designed to let users manage this dependency within their own projects. That’s why ADBC’s driver manager is included within the client library you install for your language: you can install and update it alongside your application’s other dependencies, without requiring a separate system-wide driver manager.

Once you’ve chosen an ODBC client library, you need a driver manager that both it and your database driver support. Windows includes one, and Linux distributions provide unixODBC through their package managers. If you’re using pyodbc on macOS, its recommended driver manager is also unixODBC, which you can install with Homebrew:

brew install unixodbc

For Snowflake on macOS, though, the driver instructions call for iODBC, a different driver manager. Before you’ve even downloaded the driver, you need to check which driver manager the application and driver can both use.

Find and install the driver

Once you’ve settled on the driver manager, you can look for the driver itself. Sometimes you can get it through Homebrew or a Linux package repository and carry on using your package manager. Otherwise, you’re off to the vendor’s download page. Snowflake’s macOS download is a disk image with an installer inside, while MySQL offers several package types, including tarballs whose libraries you copy into place. Knowing how you installed one driver’s files doesn’t tell you how the next vendor distributes theirs.

And opening the package doesn’t necessarily leave you with just one library to use. MySQL’s download includes both Unicode and ANSI drivers, so you need to understand the difference and choose the one your application should use.

Register the driver

Having the right library on disk still doesn’t mean your application can ask for it by name. For that, unixODBC needs an entry in odbcinst.ini giving the driver’s name and library path. Some installers add it for you, including Microsoft’s Linux ODBC packages. If yours doesn’t, you need to add the registration yourself, starting with finding the configuration file your unixODBC installation uses:

odbcinst -j

That command reports several configuration paths, including ones for odbcinst.ini and odbc.ini. Despite the similar names, they have different jobs. odbcinst.ini registers drivers, while odbc.ini stores data source names (DSNs): named sets of connection settings, such as which driver to use and which server and database to connect to.

With the right file identified, you can edit odbcinst.ini directly or register the driver with odbcinst. The command-line tool needs a template containing the registration details, so either way, you have to find the installed library’s path and put it in the expected format.

Now, finally, the ODBC driver manager can load that driver by name, and your application can use it through the common API. But if you need another driver, you may have to start over with another vendor’s installation instructions. Updates add another variation: a package manager may handle them, or you may need to download and run another installer. You can write setup scripts and share instructions with colleagues, but you have to account for each vendor’s procedure on each operating system your colleagues use.

Where language-specific connectors fall short

After that ODBC walkthrough, you might be thinking, “I just pip install a database package. I’ve never had this problem.” Fair enough: a Python package can make installation easy, and DB-API 2.0 gives many of those packages a familiar interface. But you still have to choose which package to depend on, and that choice ties your database access to Python’s connector ecosystem.

Other languages have their own database connector ecosystems. Go’s database/sql provides a common API but requires a separate database driver. Node.js applications commonly use database-specific npm clients, sometimes through a query builder such as Knex. These abstractions can make database access more consistent within a language. They still leave you choosing from a different set of connectors when you change languages, even if the database stays the same.

Even within Python, that choice can be confusing. For MySQL, Oracle offers its official Connector/Python, while SQLAlchemy recommends community clients mysqlclient and PyMySQL. You have to weigh compatibility and vendor support, work out who maintains each package, and decide who’s likely to keep fixing bugs when the database changes or a maintainer moves on. The scavenger hunt happens before you even try installing a connector.

The configuration can differ even between Python connectors for the same database. Connector/Python takes a top-level ssl_ca option, for example, while mysqlclient takes the certificate authority file path inside an ssl dictionary. Add another language and you have another connector’s configuration, dependencies, and release cycle to understand. Each independently maintained connector adds another piece of your software supply chain to vet and keep updated. You’re relying on its maintainers to patch known vulnerabilities (CVEs), so an unmaintained connector can become a security risk even if it still works.

That work can become a reason to stay with a language: your database connection already works there. It’s an increasingly awkward constraint now that AI coding tools make it easier to work in other languages. Even pandas creator Wes McKinney describes choosing Go and Rust for new projects because faster builds and tests let his coding agents iterate faster. That change shouldn’t force you to abandon a driver you’ve already configured and learned to trust.

There’s also the question of whether a package is what it appears to be. Someone with no connection to a database vendor can publish a PyPI package with a name like pymydatabase, and a malicious publisher can use the same naming convention. PyPI removes malicious packages, but detection and removal take time, and people can download and install them in the meantime. A plausible name alone doesn’t establish authorship, safety, or vendor endorsement. People and coding agents can both mistake that familiar naming pattern for evidence that a connector is official or trustworthy.

This brings us back to one of ODBC’s best ideas: the database driver can be shared across languages. ADBC follows that model: you install a client library for each language, but can keep the same database driver and connection settings. That leaves the problem of finding a driver you trust and getting it installed.

How we made ADBC driver installation easier

For ADBC, we wanted installing a driver to be as simple as asking for it by name. Finding the right download, installing the driver, and making it available to your client library should all happen automatically. One command should do the whole job.

That’s why we built dbc, the package manager for ADBC drivers. It installs drivers from the ADBC driver registry, a global content delivery network (CDN) serving prebuilt driver packages, code-signed and notarized by Columnar. The registry is affiliated with the ADBC Driver Foundry, the community project for developing and distributing ADBC drivers. Columnar provides the hosting infrastructure.

The registry’s maintainers know who wrote each driver and verify that its authors work for the companies they claim to represent. They choose the canonical entry for each database and coordinate with vendors whenever possible to provide the best driver. A name such as snowflake identifies that selected driver, so you don’t have to guess which package name belongs to the right project.

Some vendors haven’t engaged with the Foundry yet, so in those cases the registry offers carefully vetted drivers contributed by the community or built by Columnar. We’re actively pushing for those vendors to join the effort, and when they do, we’ll work with the Foundry to ensure that dbc installs the best official driver.

Install dbc

To get started, first install dbc itself. You have several installation options, depending on your operating system and preferred tools: shell and PowerShell installers, Homebrew, WinGet, uv, npm, pipx, and more. For example, on macOS you can use Homebrew:

brew install columnar-tech/tap/dbc

Install the driver

Once dbc is installed, installing a driver is just one command, dbc install <driver-name>, whether you’re on macOS, Linux, or Windows. For Snowflake:

dbc install snowflake

That’s it. The driver is installed and ready to use. No hunting for downloads, tracking down library paths, or registering the driver by hand. dbc installs for the current user by default; you can also install system-wide for all users.

Under the hood, dbc has selected and installed the correct driver package for your platform. The package contains both the shared library and a manifest that tells clients how to load it. Manifests serve the role of driver registrations in odbcinst.ini. Any ADBC client can then load the driver using the name snowflake: its driver manager reads the library’s path from the manifest.

Connect from your application

For example, to use that driver from Python, install the ADBC client library with pip install adbc_driver_manager. The package includes the driver manager inside your Python environment; there’s no system-level driver manager to install or configure.

Now you just need to supply your server and authentication details, either in code or in an ADBC connection profile. Profiles serve the role of ODBC DSNs. You provide the connection settings, and dbc has already taken care of installing the driver and making it available to the client.

Add or update drivers

With that first connection set up, adding another driver uses the same procedure. You can find which drivers are available with dbc search and install any available driver with a command such as dbc install postgresql. So the next database, or a colleague’s different operating system, doesn’t send you back to another vendor’s download page.

When you want to update the Snowflake driver, just repeat dbc install snowflake to get the latest release. To remove it, run dbc uninstall snowflake. Those operations follow the same pattern for the other drivers in the registry. And if imperative commands aren’t your style, driver lists let you declare the drivers and versions your project needs in dbc.toml and install them with dbc sync.

ODBC got portability right: a common API and drivers that work across languages. ADBC builds on that model, while dbc and the Driver Foundry give you a vetted driver by name, a consistent way to install and update it across operating systems, and connection settings you can keep when you change languages. We’ll keep benchmarking how much faster Arrow can move the data, but ADBC earns its place even before the first query runs.