Replayable Episode
The smallest unit of agent evaluation is not a benchmark question but a replayable episode: a small copy of real work. For code the start state is the commit before the pull request, for an incident the system at T0, for data a snapshot of the data, catalog and lineage.
An episode has five links: a frozen start state, an agent contract, a hidden judge, trajectory analysis and a release gate. Without a contract you compare random sets of privileges and context rather than agents — and a correct ending never justifies a dangerous path.
Two rules turn the model into working practice. Reliability is measured across a series of runs: pass^k, trajectory variance and the cost of a durable success instead of one lucky pass. Freshness rests on two sets — a fixed holdout for regression and a rolling set of live tasks. A practical start is a catalog of 30–50 episodes from your own history.
Take a completed task from your own history and freeze its start state so the future answer stays hidden.
Write the agent contract: allowed tools and network, read and write rights, limits on attempts, time and budget.
Make the judge executable, score the trajectory alongside the result, and run a series instead of a single pass.