Agents over data: why SQL is only the beginning

Consider a teaching example. An orders table has two records, worth 100 and 200 currency units. The first order has two partial refunds, of 20 and 30. For this example, define revenue after refunds as the sum of orders minus the sum of refunds. The correct result is 250.

The agent uses a LEFT JOIN to join orders to refunds, keeping orders without refunds, and sums both columns. The first order appears twice in the join. The order total becomes 400, refunds total 50, and the result is 350. The syntax is valid, the database is happy, and no exception is raised. The error is in the data’s grain: one order row met several refund rows.

The model can fix the query by aggregating refunds to the order level first. In a real task, however, another layer follows: should revenue use the payment date or the recognition date, should tax be deducted, and which period should contain a refund for last month’s order? Column names cannot reliably supply these rules. Sometimes the right next step is to find the metric definition or ask the data owner.

In an ordinary development task, a test often already exists alongside the code. For an analytical question, the test may still need to be defined. Writing SQL is therefore only part of the work. The agent has to establish the meaning, construct the calculation and give us grounds for accepting the result.

An agent harness connects the model to tools, state and an execution environment. I discussed the broader evolution of this environment in a separate article. Data platforms add a particular challenge: the agent works with code, a changing database state and the rules for interpreting that state at the same time.

It helps to distinguish the working agent harness from the evaluation harness. The former helps the agent complete its task. The latter provides the initial state, constrains resources, runs the agent and independently checks the outcome. If the working agent can see the evaluator’s hidden reference answer, we are measuring a different capability.

This is my decomposition of the requirements, not a mandatory list of six services. One platform can cover several functions. The test of usefulness is straightforward: which uncertainty does it remove for the agent, and which class of errors becomes visible to the evaluator?

A larger harness need not mean more step-by-step instructions. The model can retain freedom to choose its queries, investigation method and order of repairs. The surrounding environment can still enforce access, budgets, initial state and acceptance criteria. Knowledge about a particular table should not become another universal paragraph in the system prompt when it can be retrieved from an up-to-date catalog.

The first approach is a built-in agent. According to its documentation, Databricks Genie Code works with notebooks, SQL, jobs and other platform surfaces, using documentation, Unity Catalog and workspace code. Snowflake Cortex Code (CoCo) integrates with Snowflake schemas and its role-based access model. The potential advantage lies in a shorter path from the question to the relevant context and action.

The second approach is a familiar coding agent connected to data tools. For example, dbt MCP exposes models, metrics, data lineage and freshness, along with supported actions. Depending on configuration, it runs as a local server or a managed remote connection. MCP itself does not choose the correct metric or create an access policy; it provides an interface through which the client can use those capabilities.

The third approach is a custom environment around the model: a catalog, SQL/Python execution, session state, constraints and a verifier. Building one makes sense when domain rules or the need to span several platforms fit poorly into a ready-made integration. The price is responsibility for task isolation, adapter updates, tracing and reproducibility.

These approaches can overlap. A built-in agent can call external tools; a general-purpose agent can use a semantic layer. The more useful way to choose is to trace a concrete task from start to finish. Does the system have the required metric definition? Can it inspect a failed job? Does it receive only the rows it is allowed to see? How does it establish that a repair worked? Counting integrations does not answer these questions.

In Snowflake’s comparison, CoCo and Claude Code using the same Opus 5 model achieved success rates of 73.8% and 69.6%, respectively; average cost per attempt was $0.756 and $2.959. That is roughly 4.2 percentage points in quality and a cost ratio of about 3.9 to one. The interesting result is the simultaneous improvement in quality and cost.

A footnote adds a material condition. In the August 3–5, 2026 runs, CoCo had native access to Snowflake schemas and metadata, while the other agents used standard SQL interfaces; settings remained at their defaults. A common model does not make the tools identical. The experiment measures the combined effect of harness and integration. That is useful to a system buyer, but insufficient to isolate the contribution of the agent loop.

The same report gives another figure: CoCo solved 64.1% of tasks in all three repetitions. This is below its average success rate of 73.8%. The average share of successful attempts and success in all three runs are different properties; the latter is closer to the question of repeatability.

Databricks reports 76.6% and $0.55 for Genie Code versus 72.1% and $1.09 for the nearest competitor on 401 internal tasks with a 20-minute limit. This comparison uses models from the same frontier class, rather than an identical model. The coding agents ran in their own harnesses, but every participant was connected to Databricks MCP. Competitors are anonymized and the dataset is private. Genie Ontology was also disabled in the experiment, so the observed gain cannot be attributed to it.

Both reports give reason to test a specialized environment on your own work. Neither establishes a universal winner. Comparing complete products and isolating a component’s contribution require different experiments; the phrase “the same model” can easily conceal that distinction.

No single dataset covers all data work. Repairing a dbt project, analysing payment files, clarifying a user’s question and operating a cloud platform’s interface demand different capabilities. Strong results on short SQL queries tell us little about recovering a failed job.

In the table, “local” means that you can control the data, execution and evaluation. The model may still be accessed through an API. A fully self-contained run additionally needs a compatible local model, sufficient hardware and no mandatory external services. Being able to download tasks does not imply access to their reference answers.

For an initial comparison, I would use data-eng-bench, ADE-bench and DataSpace’s 60 open tasks. The first provides verifiable changes to data state, the second is convenient for measuring the contribution of skills, and the third extends the work beyond a single SQL dialect and file type. Report their results separately: averaging the three percentages would hide the differences between their tasks.

The dataset version is part of the result. For example, Data Agent Bench v3, dated October 7, 2026, contains 104 queries, 17 datasets and four database engines; the early version contained 54 queries. Leaderboard configurations also differ in their hints and tuned prompts. A benchmark name without a revision and run configuration does not define a comparable experiment.

Start with the question the experiment must answer. To choose a product, compare its standard capabilities: native tools, context retrieval and session management. To isolate the harness contribution, hold the available data, tools and permissions fixed, then change one component. If the systems cannot be configured that way, the result remains a comparison of systems.

DataSpace offers a useful example of experimental discipline. Its shared environment assigns the same model to both the main agent and auxiliary calls: subagents, history compression, summarization and classification. Fallback switching is disabled, and available traces are checked. Each harness nevertheless retains its own planners and tool interfaces. This brings the conditions closer together without turning different harnesses into one algorithm.

Reproduction also requires pinning the version of that environment. Historical results from before the move to Data Workbench Runtime 1.0 cannot simply be pooled with new ones. Likewise, a common model provider does not guarantee the same weight snapshot, request handling or reasoning mode.

What to record for each run

1. The model and every call to it. The exact identifier, provider, generation parameters, output limits, auxiliary models and actual values recorded in the trace. If the provider does not guarantee an immutable snapshot, state that limitation.
2. The environment. Dataset and harness revisions, container image, database snapshot, schema and document versions, time zone, permissions, network access and available tools. Each attempt starts from a clean state.
3. The budget. The same time limit and predefined cost constraints. Record step counts separately: one step can hide different numbers of model calls and queries in different agents.
4. Independent verification. The agent can access the task specification and working checks; reference answers and final tests remain outside its environment. Restrict internet access so it cannot download published solutions.
5. Repetitions and failures. The same tasks for every configuration, several independent attempts, and an alternating run order. Track infrastructure failures, incorrect answers and budget overruns separately.

The CoCo adapter instructions for data-eng-bench are particularly instructive: a YAML model_name value can override the model passed on the command line, while the adapter accepts reasoning_effort without applying it. Support for tool-disabling parameters also depends on the Harbor version. The documentation ties these caveats to Harbor 0.22.0 and CoCo 1.1.66. Commands that look symmetrical do not prove that the runs are symmetrical.

Tune the harness on a designated part of the dataset, then run the final evaluation on held-out tasks. Thirty tasks with three repetitions are thirty scenarios, not ninety independent scenarios. Repetitions reveal randomness in behaviour; they do not eliminate uncertainty about transfer to other tasks. For the final comparison, report the paired difference in outcomes and an uncertainty interval across tasks, along with the failure trajectories themselves.

The cheapest attempt can produce expensive work. Someone has to notice the wrong grain, check the figures, rerun the calculation and explain the discrepancy. If nobody notices, the cost moves into the decision made from the incorrect table.

Cost per accepted result = costs of all attempts, computation, verification and corrections / number of accepted results.

This is a proposed management metric, not a benchmark standard. Calculate it on the same set of tasks with the same acceptance criteria. Failed attempts stay in the numerator; if no results are accepted, the metric is undefined. Alongside it, report success rate, consistency across repetitions, median and tail latency, platform compute costs and human effort. A single average hides too much.

For a task that modifies data, acceptance includes database state and side effects, not just the agent’s final text. If a repaired dbt model builds the right table but also affects a neighbouring schema, checking one answer will miss that. For an analytical question, useful evidence includes the number’s provenance, the metric definition used and explicit assumptions.

Public datasets help eliminate weak configurations, but cover the specifics of a company poorly. An internal suite should add cases with outdated documentation, late-arriving events, row-level permissions, changed schemas and unclear business metrics. For some of these tasks, the correct outcome is a clarification or a justified stop. An evaluator that rewards only producing a number will encourage confident invention.

The first step is to check the evaluation setup itself. ADE-bench provides sage, an agent that uses a reference solution without calling a model. Then use a small sample to confirm that the database resets, every model call is visible, constraints work, and the final evaluator actually catches an incorrect output. Otherwise, an attractive chart may describe quirks of the runner.

Next, I would choose three available harnesses that can use a common model. If one does not support the endpoint or mode, move it into a separate product comparison. For initial selection, use fast-30 in data-eng-bench with three attempts per task. Run the final evaluation on the remaining tasks with fixed settings. If needed, report the result for the full dataset separately, identifying the subset used for tuning. Add DataSpace’s open subset to check transfer to a different kind of work.

In a separate ADE-bench experiment, keep one harness and one model, comparing no additional skills with the dbt skill set. That is a clear test of one hypothesis. An MCP experiment must account for the supplied set’s Snowflake limitation; it cannot be described as fully local with DuckDB.

Finally, add your own tasks with known business rules and hidden evaluation. First define the tolerable error and the cost of human oversight, then see which configuration meets those conditions. If the advantage disappears once repetitions, computation and corrections are counted, that is a useful result too.

A data platform already holds much of the knowledge an agent needs to find: its catalog, definitions, permissions, dependencies and execution history. A good harness makes that knowledge available at the right moment and leaves a trace that can be checked. When choosing one, I would therefore ask to see the path from a question to an accepted table—including the point where the system discovered that its first answer was wrong.