Agent evals: telling whether an agent succeeded

An early team can make progress through manual checks, using its own product, and collecting feedback. Anthropic describes Claude Code developing this way before adding evaluations for file editing, concision, and more complex behavior such as over-engineering. As usage grows, intuition becomes insufficient. A complaint that the product has deteriorated does not explain what changed or establish whether the problem is reproducible.

The outcome is the state of the environment: a modified file, an actual reservation, or satisfied task conditions. The trajectory records actions, tool calls, and intermediate responses. It helps explain failure, but a convincing final message cannot substitute for inspecting the result. In the MCP-server example, reading documentation and running tests are observable steps; the resulting implementation must still satisfy the task.

An evaluation harness runs an agent comprising a model, tools, context management, planning, and constraints. The surrounding evaluation system executes tasks, records trajectories, invokes graders, and aggregates results. Comparisons concern pinned configurations of that complete system. Changing the model, instructions, or context-compaction strategy can alter behavior even when other components remain unchanged.

A suite contains tasks with inputs and success criteria. Each task can have multiple trials and graders. The host illustrates two ways to spend one hundred runs: twenty tasks attempted five times reveal variation within tasks, while one hundred different tasks attempted once explore broader coverage. These designs answer different questions. For a nondeterministic agent, one successful run does not establish consistency, and equal run counts do not imply equal coverage.

Code-based checks are fast and useful when a condition can be expressed precisely. They can become brittle when several outcomes are acceptable. A model judge can assess open responses, clarity, or explanations, but adds its own variability and needs calibration. Human experts help validate the grading system, although they require time, money, and sometimes scarce expertise. The practical sequence is code for formal conditions, models for other necessary criteria, and people to check evaluation quality.

A single score often hides useful information. A support agent might identify the problem and verify the customer but fail to complete a refund. Partial credit exposes that boundary. Token consumption, tool calls, and elapsed time provide additional diagnostic evidence. Dialogue agents also require assessment of the interaction; research agents need grounded claims, coverage, and good sources. One generic final-answer check cannot replace these distinct requirements.

The host returns to episode 26 with Evgeny Sergeev on evaluations and episode 21 on SWE-Interact and SWE-Together: dialogue with a developer changes the conditions for evaluating a coding agent. The example of an environment where the agent cannot see a dialog or hear audio continues the discussion of autonomous application development in episode 36.

pass@k asks whether at least one of k attempts succeeds. It is useful when retries are acceptable and correct results can be identified cheaply enough. pass^k requires success in every attempt and describes consistency. The latter matters particularly when errors are difficult to detect before the result is used.

In the illustration, a 75% single-trial success probability yields roughly a 98% chance of at least one success across three independent attempts, but only about a 42% chance of three consecutive successes. Independence is an assumption of the example, not an established property of real runs; the host explicitly notes this limit. A rising pass@k must therefore be weighed against retry costs and the reliability of selection. Additional answers alone do not establish which one deserves trust.

The authors advise starting without waiting for hundreds of tasks or elaborate infrastructure. An initial twenty to fifty examples can come from manual checks, support requests, and bugs, prioritized by user impact. Each needs an unambiguous specification and aligned criteria: independent experts should be able to reach the same verdict. A reference solution establishes that the task is solvable and that graders accept a valid result.

Negative examples matter too. Rewarding web search only on tasks that require it can encourage searching everywhere. Cases where search is unnecessary make the criterion meaningful. Every trial should start clean, without leftovers from other runs or access to answers through repository history. The environment must also expose the feedback an agent needs. An invisible browser dialog or inaccessible audio limits what the agent can verify.

Model-grading criteria should be separated: empathy, clarity, and compliance with company policy are different properties. Read trajectories and compare the agent’s actions with the grader’s verdict. Failure may originate in the agent, harness constraints, an unstable environment, an ambiguous task, or an incorrect reference answer. Repair the identified cause rather than merely pushing up the aggregate score.

The episode discusses excessive numerical precision and a check expecting a file at a path absent from the instructions. Such failures do not establish that an agent cannot perform useful work. Capability and regression suites also serve different purposes: one exposes tasks the system cannot yet solve, while the other protects established behavior through subsequent changes. To inform product decisions, evaluation conditions need to resemble actual use.

Pre-release checks are complemented by monitoring, experiments, support reports, and observations of user behavior. The host compares this with established software testing: nondeterminism and model judges add complications, but investigating failures and assessing changes remain familiar engineering work. A platform team can maintain the execution infrastructure while product teams own the tasks and criteria relevant to their agents.

The epilogue broadens the question of trust. Petri v2 raises the possibility of a model changing its behavior when it recognizes an evaluation. Research on autonomy examines real sessions, where users and product design also shape outcomes. Claude Code’s regression report illustrates how reasoning settings, history cleanup, and concision instructions can degrade performance. Even mature evaluations miss problems before release and must continually be compared with what users experience.

The closing recap connects the harness reviews: episode 33 on SWE-agent interfaces, 34 on work handoffs between sessions, 35 on repository environments, 36 on planner, executor, and evaluator roles, 38 on experiments with planning, tools, and context compaction, and 41 on architecture inferred from source code. This episode adds verification of outcomes to that account of system design. Links to the mentioned episodes appear below; timestamps refer to the Podster audio recording.