Open tables need open connectivity

You’ve moved your tables into Iceberg. Your data lives in object storage, and multiple engines can query it. You can choose where a workload runs without first copying its data into another warehouse. This is the architecture you wanted: open tables, a choice of engines, and nothing locking you in to any one of them.

Then you try to connect an application to a second engine.

You need another driver. It has different installation instructions, authentication options, and configuration settings. Your application needs to handle a different result format. Your platform team needs to approve another access path and figure out where its logs go. When you finally get a query running, the database says it finished a while ago, but your application is still waiting for the results.

Your engines can share tables through a common standard, but connecting your applications to those engines is still an integration project. At Columnar, we think this is one of the biggest unfinished jobs in data infrastructure. Open tables need a complementary layer of database access built on shared, open standards. And as agents take on more of the work of building applications and analyzing data, the costs of leaving that layer unfinished become harder to tolerate. We’re building the missing piece.

The other side of the engine

Apache Iceberg lets multiple engines work with the same tables. That changes the relationship between storing data and choosing the system that processes it. An engine can earn its place because it is good at a workload, while the tables remain available to other engines.

The opportunity is enormous. Any engine that reads Iceberg can prove itself on your real data, with no migration in the way. You can keep a warehouse for workloads it handles well and use another system where it makes more sense. You have more options when requirements change, a contract comes up for renewal, or your monthly bill unexpectedly spikes.

But an engine has two sides. Below it are the tables it reads. Above it are the applications that send queries and consume results. An open table format addresses the storage side. The application side still needs a consistent way to connect and retrieve results.

An application can still be tightly coupled to a vendor’s client library, result representation, and authentication machinery, even when the underlying data is in Iceberg tables. A platform team can still face a separate monitoring and security project for every engine it adds. Those costs influence architecture. The next workload goes to the engine that is already connected and approved, even when another engine would be a better fit.

We believe modern data infrastructure needs a common foundation on both sides: open tables below the engines, and open interfaces above them. Iceberg addresses the first. Apache Arrow and ADBC (Arrow Database Connectivity) provide the foundation for the second. With a common client API and result format, developers can reuse their connection and result-handling code when they add another engine. The integration work gets smaller, making it easier to put that engine to use.

The query finished. Where are the results?

Modern analytical databases work in columns, and so do many of the tools that consume their results. Yet the interface between them often follows the row-oriented model of ODBC, JDBC, or Python’s DB-API 2.0: the driver converts columns into rows, and the application converts them back. A fast database and a fast analysis library can spend much of their time waiting on the code between them.

The cost of those conversions shows up in benchmarks. In our test querying six million rows from BigQuery, the fastest ODBC wrapper we tested took about 2.7 times as long as ADBC. The benefit can be much larger: DuckDB’s 2023 data-transfer benchmark reported a 38× speedup over ODBC. In many cases, changing the client interface alone can make a substantial difference to how long an application waits.

Apache Arrow specifies a common columnar data representation, including the schema and memory layout, and over the past decade, it has become near-ubiquitous across modern analytics and AI, built into databases, query engines, dataframe libraries, and machine-learning tools. If you use a modern data stack, you are almost certainly using Arrow, often in several places and without having explicitly chosen it. Those systems can already exchange columnar data without reconstructing it value by value.

ADBC brings that shared format to a standard database access API. When a database produces Arrow, its driver can pass the results directly through to an Arrow-aware consumer. When conversion is necessary, the driver handles it in efficient compiled code before handing the results to the application.

Preserving data types matters for correctness too. A decimal has a precision and scale. An integer has a width. A timestamp has a unit and may have a timezone. Arrow carries that information with the data, giving clients a consistent type model and reducing the need to infer what an untyped string or generic object was supposed to mean.

This is why we named the company Columnar. Keeping data in the right shape saves an extraordinary amount of work.

Let agents run at agent speed

To understand why this matters for agents, consider how a human analyst works. After running a query, they spend time examining the results, thinking through possible explanations, and deciding what to ask next. If most of the analysis is spent on that reasoning, faster retrieval saves only a small fraction of the total time.

An agent can work through those steps much more quickly and be ready for the next query sooner. The time spent executing queries and retrieving results then accounts for a larger share of the task, even if the agent runs exactly the same queries and retrieves exactly the same data as the human. Reducing retrieval time has a greater effect on how quickly the analysis finishes. A connection that seemed fast enough for a human can leave an agent spending most of its time waiting.

The figure holds the number of queries constant. In practice, an agent exploring a dataset issues many more: profiling columns, testing hypotheses, checking its conclusions. In iterative work such as root cause analysis, each result decides what to investigate next, so every retrieval delay holds up the following query. Keeping results columnar lets the agent move on sooner.

The scavenger hunt starts before the query

Agents hit another connectivity problem before they ever fetch a result: getting connected. Ask a coding agent to build something against a database and it has to decide which package to use, whether it is maintained, and which authentication it supports. Change the database or the language and the choices start over. Each successful attempt may leave behind another integration for your team to maintain.

We know this problem well. We install and configure competing drivers as part of our benchmarking work, and recently wrote about the experience. Finding the right download and locating its shared library are only part of it. You may also need to install a system-wide driver manager and edit configuration files before your application can load the driver. Language-specific packages often make installation easier, but leave each language with a separate connector ecosystem to evaluate.

ODBC got an important idea right: a common database access API, implemented by drivers that applications can use across programming languages. ADBC carries that idea forward. Its client libraries span the major programming languages: C/C++, C#, Go, Java, JavaScript/TypeScript, Python, R, Ruby, and Rust. The same driver can be used from all of them, so changing languages doesn’t mean finding a new database driver.

We built dbc to make those drivers straightforward to install and manage. For example:

dbc install snowflake

That command installs the right package for your platform and makes the driver discoverable by ADBC clients; add your connection settings and you can start running queries. The drivers behind it are tested, tuned for performance, and maintained as databases evolve, and through the ADBC Driver Foundry we partner with vendors and the open source community to make dependable drivers available and easy to install. A coding agent then spends less time choosing packages, and the applications it produces share one access model that a team can review, operate, and update.

Query demand is no longer gated by human pace

Query demand has traditionally followed the pace of human analysis and software development. Automated jobs can already execute at enormous scale, but new sources of demand have generally emerged as people investigate questions and build reports, pipelines, and applications.

Agents remove many of those natural gates on query demand. The queries they issue are unpredictable in both volume and variety. They can pursue investigations in parallel, run rapid sequences of queries in which each result informs the next, and build new tools that keep issuing queries after the initial investigation is over. The same team and user base can produce a much larger and more varied set of workloads without waiting for people to initiate each investigation or build each tool. Query volume, complexity, and concurrency become much harder to anticipate: a forecast based on headcount, user count, and the applications already in production no longer tells you what demand to expect.

That undermines the economic model as well. A warehouse budget that worked for a known set of teams, users, and applications can become untenable as agents create new sources of demand. Some teams already face similar pressure in operational analytics, where query demand scales with user count rather than employee headcount. Agents amplify that pressure even without user growth. Optimizing today’s queries does not account for the workloads agents may create tomorrow. If all of that activity inherits the same warehouse as its default, the compute bill can become a limit on how much useful analysis agents are allowed to do.

Iceberg gives teams the option to put those changing workloads on different engines over the same tables. DuckDB can read Iceberg tables directly, providing a potentially much cheaper option than a cloud warehouse for suitable single-machine workloads. Keep the warehouse for workloads that benefit from it, while making lower-cost engines available for others. The savings depend on the workload and deployment, but the ability to change where queries run matters much more when you cannot predict which workloads will appear next.

Connection sprawl becomes a production risk

Each new tool needs a managed way to reach the engines it uses. In a production application, each connection needs to be maintained, monitored, and secured for as long as the application is running. A stale driver, exposed credential, or unmonitored access path can put data at risk.

As the number of applications and databases grows, those risks become harder to track. A patch for a critical vulnerability may be applied in one service while others continue loading the vulnerable driver. Credentials may outlive the jobs they were created for. Different teams may configure access differently, with no shared record of which applications can reach which data. Each independently managed connection creates another place for an update or policy change to be missed.

Platform teams need to trace each request to the application and identity that made it, see which data it queried, and observe how much data it returned. They need an audit trail for investigating suspicious activity and a way to revoke the access involved.

Fragmented connectivity makes that work harder across the whole stack. An engine that is convenient for an individual developer or team can be difficult for the organization to operate. A useful DuckDB experiment can become a scheduled job and eventually power a feature customers rely on, before anyone has decided how its access should be managed.

Faced with that burden, a company may keep every workload inside one data warehouse vendor’s platform, where access controls and monitoring are already integrated. An engine outside that platform then has to justify the extra work of setting up and operating another access path. That is one way a vendor’s walled garden can remain hard to leave even after the data moves to open tables.

We’re building Columnar Gateway to bring connection management, security, and observability together across vendors. It is a unified access layer for applications and agents, built on Arrow and ADBC, and is in private preview with select design partners. Application teams get a consistent way to reach their databases and query engines; platform teams get a common place to monitor and control those connections.

Managing drivers and connections in Gateway lets a platform team update a driver or change an access policy in one place, applying the change to the connections it manages. Application teams use those managed connections without each having to install the update or implement the policy themselves.

Managing connectivity behind a service boundary also lets teams isolate driver runtimes from their applications. A native driver loaded inside an application shares its process, so a driver crash can bring the application down with it. Running the driver outside that process allows failures and updates to be handled there. We want teams to have that deployment choice while keeping a familiar database access interface.

Make engine choice real

This brings us back to Iceberg. If several engines can work with the same tables, the next question is whether your applications can make practical use of them. A common access layer reduces the integration and operational work required to try another engine, move a workload, or introduce a new tool.

That changes the economics of a data stack. An experiment can get a supported path to the data it needs. The warehouse that won last year’s evaluation can compete for the next workload on its merits. The option to move becomes more credible when doing so involves less work above the engine as well as below it.

Reducing vendor dependence is one part of digital sovereignty. Organizations need the practical ability to move workloads, adopt new engines, and change providers while retaining control over data access. Open tables and open connectivity make those decisions easier to act on.

We want teams to be able to choose where they store data, which engines query it, and how they build applications without turning every change into an integration odyssey. Iceberg provides an open format for organizing data files into tables that multiple engines can read and update. Arrow gives analytical systems a common representation for exchanging data. ADBC gives applications a common database access interface. Drivers and dbc make that interface usable in practice. Gateway gives teams a place to manage those connections across their stack.

The benefits extend to databases outside an Iceberg architecture. ADBC drivers can return Arrow from transactional systems, converting the results for analytical use. A team can use ADBC to load PostgreSQL results into a dataframe today, then reuse that access model with a warehouse or an engine querying Iceberg. Existing databases and new engines can be served through the same connectivity layer.

What a database connection should provide

When we announced Columnar, we described the need to bring speed, simplicity, and security to database access. The more of this infrastructure we build, the more closely connected those goals become. A fast interface needs drivers people can install and depend on. Freedom to choose an engine needs a practical way to govern access to it. Platform teams need to see which applications are querying which databases, under whose authority, and with what performance.

Arrow and ADBC are developed under the Apache Software Foundation and are open for anyone to use and build on. We’re investing in those standards and their drivers and tools while building Gateway on the same foundation.

Open tables give us a choice of engines. Open connectivity lets applications exercise it, and as agents take on more of the work, the speed of those connections matters more than ever. We’re building Columnar to make database access fast, simple to use, and secure to operate across your entire data stack.