
Agents over data: why is SQL not enough?
Metric definitions, data lineage and access rights, plus a fair same-model comparison of harnesses
Slide contents
1. Agents over data: why is SQL not enough?
Metric definitions, data lineage and access rights, plus a fair same-model comparison of harnesses
2. The SQL ran, yet revenue is inflated
350 instead of 250: one order met two refunds
3. Column names do not carry metric meaning
Sometimes the right step is to find a definition or ask the owner
4. From a wrong number to fair comparison
Harness, evidence, benchmarks, protocol
5. A harness is not an evaluation rig
One helps do the task, the other checks the result independently
6. Context, meaning, execution: what the agent gets
Metric definitions, grain and data lineage
7. Checks, authority and trace make results verifiable
Permission to read is not permission to export
8. Three ways to build a platform agent
Built-in agent, coding agent plus tools, your own environment
9. Check the task's path, not integrations
Four questions from the request to the accepted table
10. CoCo: more accurate, almost four times cheaper
73.8% vs 69.6%; $0.756 vs $2.959 per attempt
11. Same model does not mean same tools
CoCo had native metadata access; the others used standard SQL
12. Average success exceeds success in every run
73.8% of attempts, but 64.1% of tasks in all three runs
13. Databricks compares systems, not one model
76.6% and $0.55 vs 72.1% and $1.09 on 401 tasks
14. Two questions need two different experiments
Comparing products and isolating a component are different tasks
15. No single benchmark covers all data work
Eleven benchmarks: dbt, payments, clarifications, interfaces
16. Three benchmarks are enough to start
data-eng-bench, ADE-bench and DataSpace's 60 open tasks
17. One model means every call to it
Subagents, history compaction, summarization and classification
18. A run passport has five sections
Model, environment, budget, independent verification, repetitions
19. Symmetric commands do not prove symmetric runs
What the CoCo adapter does with run parameters
20. Thirty repeated tasks are thirty scenarios
Tune on a dedicated split, report on held-out tasks
21. Count the cost per accepted result
A cheap attempt can turn into expensive work
22. Start the experiment by checking the rig
The rig, three harnesses, one component, your own tasks
23. The numbers belong to the report authors
Source snapshot: November 13, 2026
24. Ask for the path to accepted tables
And the point where the system caught its own first mistake