Replayable Episode
- One commit or snapshot
- Same data and services
- The future answer is hidden
- Allowed tools and network
- Read, write, approve
- Attempts, time, budget
- Code: hidden tests and policies
- Workflow: tool calls and arguments
- Data: state and invariants
- The right tool was chosen
- Arguments match the contract
- No dangerous side effects
- Accepted or not
- With evidence, not persuasion
- The same bar every version
- Static holdout — regression gate
- Rolling live set — fresh work
The smallest unit of agent evaluation is not a benchmark question but a replayable episode: a small copy of real work. For code the start state is the commit before the pull request, for an incident the system at T0, for data a snapshot of the data, catalog and lineage.
An episode has five links: a frozen start state, an agent contract, a hidden judge, trajectory analysis and a release gate. Without a contract you compare random sets of privileges and context rather than agents — and a correct ending never justifies a dangerous path.
Two rules turn the model into working practice. Reliability is measured across a series of runs: pass^k, trajectory variance and the cost of a durable success instead of one lucky pass. Freshness rests on two sets — a fixed holdout for regression and a rolling set of live tasks. A practical start is a catalog of 30–50 episodes from your own history.
Take a completed task from your own history and freeze its start state so the future answer stays hidden.
Write the agent contract: allowed tools and network, read and write rights, limits on attempts, time and budget.
Make the judge executable, score the trajectory alongside the result, and run a series instead of a single pass.