How to Evaluate AI Agents: From a Convincing Answer to a Reproducible Engineering Episode

The longer you study modern benchmarks and industry practice, the clearer one thing becomes: pass rate no longer guarantees anything. An agent can pass every test and still leave behind a dangerous diff. A correct answer may appear only because the agent saw the future solution. It may solve a task on the first run and head in a completely different direction on the second. It can produce architecturally convincing prose that cannot be turned into a single team task, or write SQL that returns a number while joining tables at the wrong grain.

The key shift for me is that mature agent evaluation cannot be reduced to a leaderboard. It is an engineering quality system. The things to measure are not isolated questions and final answers but complete, reproducible episodes of work.

A good agent eval case is a small frozen copy of real work. It has an initial state, an agent contract and allowed tool set, a hidden judge, an action trace, and a measurable objective.

This pattern is already close to standard in coding benchmarks. SWE-bench takes real GitHub issues and asks whether a model can repair the repository so that the tests pass. FEA-Bench moves the focus from bugs to feature implementation: not merely fixing a defect but implementing a feature from a real PR trail. That matters because production work almost never looks like an isolated exercise. It is usually an issue plus the entire repository context: constraints, tests, local commands, review, and several iterations on top.

Even that is not enough. We need to record not only the final patch but the complete trajectory that produced it. In its offline evaluation of GitHub MCP Server, GitHub evaluates tool choice and argument correctness separately. This sounds like a small technical detail, but it is where much of the risk hides: an agent may call the wrong tool, use dangerous parameters, guess a correct answer accidentally, or spend ten unnecessary steps on a trivial task.

I would describe the minimum eval episode like this:

episode_id: "feature-checkout-discount-014"
start_state: "repo commit, tickets, docs, fixtures, test environment"
input_bundle: "business goal, acceptance criteria, constraints"
agent_contract: "allowed tools, permissions, time budget, write policy"
hidden_judge: "tests, policy checks, expected state, review rubric"
trace_checks: "tool choice, arguments, unsafe calls, unnecessary steps"
release_gate: "pass threshold, repeatability, cost ceiling, safety stop"

Several properties behind this skeleton determine whether the episode holds together.

First, the initial state must be frozen. What that means depends on the domain: a pre-PR commit for code, the system state at the first alert for an incident, or a complete snapshot for a data platform— data, catalog, lineage, and model versions. For architecture, the starting bundle contains requirements, the current system, company rules, and the list of managed services that may be used.

I always keep the agent contract explicit. What can it read or modify? Which tools may it use? Does it need human approval? Is the network available? How many attempts does it get? Without these answers, we are evaluating a random privilege set rather than an agent.

The judge remains hidden. For code, it includes hidden tests, the full suite, and static and security checks. For workflow agents, it covers expected tool calls and the final state. For architecture, it includes policy checks, solution completeness, traceability, and downstream usefulness. The more of the judge we can execute, the less room remains for polished but unsupported rhetoric.

Even an executable judge is worthless without repeatability. τ-bench is valuable precisely because it focuses on reliability across a series of runs: an agent that can sometimes solve a task may still be a poor production tool. One successful run is not quality. We need repeated runs, variance, budget sensitivity, trajectory stability, and cost.

Freshness is the final requirement. Static public datasets are quickly contaminated: tasks leak into training data and accumulate wrappers, hints, and implicit tuning. That calls for time splits, project splits, and a live rolling set. SWE-bench Live applies this idea to coding tasks, while Microsoft’s work on continuous benchmark generation states the broader principle for enterprise agents: a benchmark must evolve with the working environment.

Code is the easiest place to start in the SDLC because it is the easiest place to build an executable judge. But an agent claiming autonomy in software delivery must be measured across the entire path from requirements to operations.

Start with requirements. The unit of evaluation is not a command to “write a PRD” but a traceability chain: business brief, discovery notes, stakeholder conversations, approved requirements, acceptance criteria, NFRs, and what is eventually built from all of them. A good judge examines completeness, correctness, testability, hallucinated requirements, NFR coverage, and whether the result is useful to the next agent or a human team.

Feature coding is closest to FEA-Bench: an issue or specification, a pre-PR commit, hidden functional and integration tests, and a safety check across the full suite. Bugfixes are subtler. The case needs a buggy commit, issue, stack trace, or failing behavior, and the judge must prove that the defect reproduced and disappeared without breaking adjacent scenarios. Replacing quality with one public regression test is dangerous: a weak oracle almost inevitably accepts incorrect patches.

Code review is a separate problem. We should measure whether the agent finds defects that affect merge quality, not whether its comments sound polished. SWE-PRBench builds this evaluation around real PRs and human-annotated findings. Inside a company, it is even more useful to turn some review findings into executable checks: once a concern can be captured as a test, policy check, or static rule, it stops being a matter of taste.

Test generation maps naturally to the fail-to-pass principle: the agent must write a test that fails before the fix and passes after it. Microsoft’s TestExplora uses exactly this structure. Coverage alone is not enough for ordinary unit-test generation; the judge also needs compile and pass rate, branch coverage, mutation score, and the ability to catch real defect classes.

Incidents and SRE are the hardest. An agent cannot be given a single question such as “what happened?” It needs an incident bundle: the T0 alert, topology, all telemetry—logs, metrics, and traces—plus runbooks, the timeline, the postmortem, and a list of actions it may perform. AIOpsLab advances this kind of environment for cloud-agent evaluation. The judge must inspect every step: how the agent triaged the event, estimated blast radius, reached the actual root cause, selected safe remediation, and—most importantly—handed control back to a human safely.

An SDLC eval therefore asks one final question: did the agent carry the work episode to a state that can actually be verified?

Architecture is more difficult. There is almost never one correct answer. A good solution usually belongs to a family of acceptable options, each with its own trade-offs and hard negatives. Comparing an agent’s result with a single gold diagram is therefore an inherently weak model.

The public benchmark market here is still young. A systematic literature review of LLMs in software architecture found only 18 relevant studies and highlighted weak coverage of cloud-native systems, architecture, and conformance checking. The authors position ArchBench as a unified platform for architecture tasks: generating ADRs, recovering traceability links, assembling serverless and dynamic components, and deriving microservices directly from requirements. R2ABench is closer to requirements → architecture: real PRDs, expert PlantUML diagrams, structural graph metrics, multidimensional scoring, and antipattern detection.

Once the scenario becomes “feature idea → target architecture above the repository level → downstream work items,” an open benchmark is almost certainly insufficient. External datasets make useful seeds, but the main eval set must come from internal decisions.

The complete architecture package should be evaluated. Its input contains:

• —the feature idea and business objective;
• —product NFRs and constraints;
• —the current architecture;
• —the catalog of approved managed services;
• —organizational guardrails and security/compliance requirements;
• —the domain vocabulary;
• —integration constraints;
• —current ADRs and reference patterns.

The output is not one artifact but a package: target architecture and ADRs, service boundaries and data/event flows, platform-service choices, rollout and rollback plans, observability and security plans, and a decomposition into downstream work items.

A three-layer judge works well here. The bottom layer contains hard policy checks: approved managed services, data residency, auth patterns, PII boundaries, SLO/SLA, ownership, and mandatory observability controls. Above it sits structural quality: completeness, antipatterns, requirements-to-components traceability, quality-attribute coverage, trade-offs, and sensitivity points. The top layer is downstream executability: can the decision be decomposed into repository-level tasks that can be implemented and verified?

One mistake is easy to make here. Store a reference solution family—required elements, acceptable alternatives, and explicitly forbidden options—rather than one canonical answer. That is much closer to a real architecture review than comparing against one canonical picture.

The data-platform picture is more mature. Benchmarks already cover open-ended analysis, multi-step analytics, ELT, governance, and full-lifecycle data intelligence. If I had to choose one external starting point, it would be DAComp, because it separates repository-level data engineering from open-ended data analysis: DE-Architecture, DE-Implementation, DE-Evolution, and analytical tasks involving reports, insights, and recommendations.

The most common analyst scenario is a business question over a semantic layer or several databases: “why did this metric fall?”, “which segment declined?”, or “how many customers were affected?” The judge should focus less on the final number than on how it was obtained: correct joins, source selection, the tool trace, reproducible SQL/Python, and consistent provenance. DAB usefully formalizes production data-agent pain: multi-database integration, broken join keys, extracting data from unstructured text, and domain knowledge.

Exploratory analysis is different. The agent receives data and must produce an insight memo, charts, and recommendations. Often there is no deterministic answer. The judge therefore examines grounding in data, calculation correctness, coverage of essential analytical steps, artifact reproducibility, and recommendation quality. BLADE provides a useful pattern: open-ended analysis with expert ground truth that allows for multiple valid approaches.

A third and particularly awkward case almost always remains internal: semantic-layer conflict resolution. The agent must choose between competing definitions of one KPI, explain the discrepancy, show the dashboard impact, and propose changes to the semantic layer or documentation. Public data is scarce because these conflicts depend on company-specific business logic. The best cases live in the analytics team’s own history: incidents, postmortems, Slack discussions, and Jira tickets.

Data engineers need a completely different set. Business requirement → data blueprint starts with grain and key entities, then keys and data contracts, with lineage, the DAG, and platform mapping on top. Next comes building an ELT/dbt pipeline from a specification, where correct models, tests, contracts, and hidden regression checks matter. Evolution is another task: change an existing pipeline without breaking downstream reports. Governance adds cleaning, standardization, and policy compliance. Lineage impact analysis must estimate blast radius across upstream and downstream dependencies.

ELT-Bench is a useful reality check. Its authors assembled 100 pipelines with hundreds of source tables and models, and even strong agents correctly generate only a small fraction of those models in their measurements. That is not a reason to abandon data agents, but it is a reason never to release them without hidden checks: schema, row counts, value distributions, invariants, lineage, and proof that downstream reports still work.

For governance tasks, DataGovBench is useful not only for its cleaning and standardization tasks but for how it frames the problem. A company should evaluate a data agent not by how fluently it explains a data problem, but by whether the data actually improved without violating policy.

With scenarios this different, it is tempting to collapse everything into one universal score. I would resist that. Keep a common five-part scorecard and adjust its weights by domain.

Outcome. Was the task solved? Was the architecture accepted, the data answer correct, the table built properly, the test suite green, the incident triaged, the review finding accepted, and policy obeyed? This is the foundation. If it fails, the other metrics do not matter.

Trace. Was the plan appropriate? Were the right tools called with the right arguments? Were there unsafe or unnecessary actions? Did the agent access a future answer it should not have seen? This is the layer that distinguishes an agent eval from ordinary text scoring.

Operations. What did it cost and how long did it take—latency, cost, token burn, build minutes? How many tool calls and retries occurred? Is performance stable from run to run? An agent that solves a task at unpredictable cost and with huge variance can be fascinating in research and unusable in production.

Safety. Permission violations, PII access, touching the wrong files, unsafe remediation, policy bypasses, unnecessary writes, and the absence of a rollback plan. For SRE, security, data platforms, and architecture, these must be stop conditions rather than small penalties in a combined score.

Human acceptance. Was the PR accepted? Were review comments adopted? Did the architecture pass the board? Did the RCA help the on-call engineer? Did the analyst validate the conclusion and the data owner approve the remediation? However autonomous an agent may be, a person ultimately accepts the PR.

Industry practice is converging on roughly the same shape. Snowflake’s Cortex Agent Evaluations describes Goal-Plan-Action scoring with tool selection, tool execution, and answer correctness over traced runs. Databricks’ coSTAR uses a Scenario → Trace → Assess → Refine loop and runs the same evals during development, in CI/CD, and on sampled production traffic. That is the right form: evaluation is not separate from operations; it becomes the regression loop of operations.

Put all of this into one working format and the result is a scenario family × input bundle × hidden judge × metrics × release gate matrix.

This table matters more than choosing a particular model. A model can be replaced at any time. Without a scenario catalog, internal input bundles, hidden judges, and release gates, every model change restarts the argument about whether the agent improved.

These are the failure modes that most often break an eval program.

The most common mistake is evaluating only the final answer. An autonomous agent can produce a correct result through a completely unacceptable trajectory. If it reached the solution through git log, an external PR, overly broad SQL access, or unsafe remediation, the outcome is not a success, however polished it looks.

The second trap is trusting one gold answer. Code sometimes has a clear oracle, but architecture, analytics, and incident response usually admit many valid solutions. These domains need reference solution families, rubrics, and hard negatives.

Stale datasets are another problem. An old benchmark gradually changes from a measurement instrument into teaching material. The remedy is a static holdout for regression plus a live rolling set of fresh tasks.

Treating vendor claims as independent validation is another failure. Vendor statements about accuracy gains, time savings, or agent-platform quality are useful directional signals, but they are claims, not evidence for your environment.

Finally, offline and online evaluation must not be confused. Offline eval is a fast release gate. Online evaluation reveals the real effect: whether PRs are accepted, incident investigation gets faster, rework falls, and teams trust agent answers. Each captures only half the picture on its own.

I would not start by choosing the best model. First, assemble a catalog of 30–50 internal episodes: several feature PRs, a few bugfixes, some review cases, a handful of requirements and data tasks, two or three architecture initiatives, and a couple of incidents. Reconstruct the initial state for every episode, hide the final solution, describe the agent contract, and make the judge as executable as the domain allows.

Then create two sets. Keep a static holdout untouched and run it for regression across agent versions. Regularly refresh a rolling live set with new PRs, incidents, analytics requests, and architecture decisions. Track all five metric layers from the start: outcome, trace, operations, safety, and human acceptance.

The complete pipeline reduces to one chain:

historical work artifact → frozen initial state → agent contract → hidden judge → action trace → repeatability → release gate

This is less exciting in a demo than a story about an agent doing everything by itself. But a catalog of frozen episodes is exactly what turns an agent from an impressive demo into a tool that can be trusted in production.