Skip to content
Research Insights Made Simple logo
Live · November 16, 2026Research Insights Made Simple

Agents over data: why is SQL not enough?

Metric definitions, data lineage and access rights, plus a fair same-model comparison of harnesses

/ Research Insights Made Simple · Agents over data

Slide contents

  1. 1. Agents over data: why is SQL not enough?

    Metric definitions, data lineage and access rights, plus a fair same-model comparison of harnesses

  2. 2. The SQL ran, yet revenue is inflated

    350 instead of 250: one order met two refunds

  3. 3. Column names do not carry metric meaning

    Sometimes the right step is to find a definition or ask the owner

  4. 4. From a wrong number to fair comparison

    Harness, evidence, benchmarks, protocol

  5. 5. A harness is not an evaluation rig

    One helps do the task, the other checks the result independently

  6. 6. Context, meaning, execution: what the agent gets

    Metric definitions, grain and data lineage

  7. 7. Checks, authority and trace make results verifiable

    Permission to read is not permission to export

  8. 8. Three ways to build a platform agent

    Built-in agent, coding agent plus tools, your own environment

  9. 9. Check the task's path, not integrations

    Four questions from the request to the accepted table

  10. 10. CoCo: more accurate, almost four times cheaper

    73.8% vs 69.6%; $0.756 vs $2.959 per attempt

  11. 11. Same model does not mean same tools

    CoCo had native metadata access; the others used standard SQL

  12. 12. Average success exceeds success in every run

    73.8% of attempts, but 64.1% of tasks in all three runs

  13. 13. Databricks compares systems, not one model

    76.6% and $0.55 vs 72.1% and $1.09 on 401 tasks

  14. 14. Two questions need two different experiments

    Comparing products and isolating a component are different tasks

  15. 15. No single benchmark covers all data work

    Eleven benchmarks: dbt, payments, clarifications, interfaces

  16. 16. Three benchmarks are enough to start

    data-eng-bench, ADE-bench and DataSpace's 60 open tasks

  17. 17. One model means every call to it

    Subagents, history compaction, summarization and classification

  18. 18. A run passport has five sections

    Model, environment, budget, independent verification, repetitions

  19. 19. Symmetric commands do not prove symmetric runs

    What the CoCo adapter does with run parameters

  20. 20. Thirty repeated tasks are thirty scenarios

    Tune on a dedicated split, report on held-out tasks

  21. 21. Count the cost per accepted result

    A cheap attempt can turn into expensive work

  22. 22. Start the experiment by checking the rig

    The rig, three harnesses, one component, your own tasks

  23. 23. The numbers belong to the report authors

    Source snapshot: November 13, 2026

  24. 24. Ask for the path to accepted tables

    And the point where the system caught its own first mistake