Agent evals: telling whether an agent succeeded

Alexander opens with a practical question: what establishes that an agent has completed its work? Anthropic’s account of Claude Code describes manual testing and feedback gradually being complemented by evaluations of editing, concision, and over-engineering. The host distinguishes outcomes from trajectories. Outcomes are checked in the environment: does the booking exist, was the file changed, and are task conditions satisfied? A trajectory shows how the agent got there and helps explain failure. The evaluation harness executes a complete system with tools and configuration, not an isolated model. Twenty tasks attempted five times versus one hundred tasks attempted once illustrate consistency versus breadth of coverage.

The discussion then examines three kinds of graders. Code handles precise conditions, model judges assess open responses and qualitative criteria, and people supply expert judgment and calibration. Each has limitations: programmatic checks can be brittle, models add nondeterminism, and experts are costly and sometimes unavailable. A support workflow illustrates partial credit: identifying a problem, verifying a customer, and issuing a refund may have different outcomes. Telemetry helps locate failure. Research tasks require source quality and coverage; dialogue tasks require assessment of the interaction. The host also notes that an agent cannot confidently inspect parts of the environment it cannot access, such as audio or a browser dialog.

The distinction between pass@k and pass^k connects metrics to product scenarios. The former counts at least one success; the latter requires success in every trial. The numerical example using a 75% success probability assumes independent runs, a limitation the host explicitly states. The practical recipe starts with twenty to fifty real tasks, clear conditions, and a reference solution. It adds negative examples, clean environments, and protection against finding answers in repository history. Trajectory review can reveal faulty evaluation: unjustified precision, an unspecified file path, or unstable checks. A suite for developing capabilities is complemented by regression checks protecting established behavior.

The closing discussion connects pre-release evaluations to monitoring, experiments, and user feedback. The host compares this with conventional testing and separates platform-owned infrastructure from product-owned criteria. The epilogue turns to later Anthropic work: Petri v2 raises evaluation awareness, autonomy research examines real sessions, and the April Claude Code regression report shows the effects of reasoning settings, history handling, and concision. No evaluation suite catches every problem before users encounter it. Timestamps follow the Podster audio version; alignment with the YouTube timeline was not measured separately.