Building evals that work
Replayable episodes, hidden judges, and a production scorecard. With Evgeny Sergeev, Engineering Director at Flo Health
Slide contents
1. Building evals that work
Replayable episodes, hidden judges, and a production scorecard. With Evgeny Sergeev, Engineering Director at Flo Health
2. Pass rate does not prove quality
Correct outcome, unsafe path
Unsafe diff passes tests
Future solution leaks into context
Rerun takes another path
Correct SQL, wrong grain
3. Evaluate the system, not the model
The model is only one part of an agent loop
what — Outcome — Actual task result
how — Trace — Tools and actions
how much — Operations — Cost and stability
4. 01. Replayable episode
Freeze the work, constraints, judge, and release criteria
5. An episode fixes the whole workflow
Start state → contract → hidden judge → trace → release gate
6. The contract fixes evaluation boundaries
Otherwise, different environments and privileges get compared
Frozen start state
One commit or snapshot
Same data and services
Future solution stays hidden
Agent contract
Allowed tools and network
Read, write, approve
Attempts, time, budget
7. The judge should execute
Persuasive language is not evidence
Code: hidden tests and policies
Workflow: tool calls and arguments
Data: state and invariants
Review: findings become checks
8. The path is part of the result
Correct output cannot excuse harm
The right tool was selected
Arguments match the contract
No unsafe side effects
No needless loops or retries
9. One success is not reliability
A run series matters more than the best run
k runs — Pass^k — Success across runs
σ — Variance — Trajectory dispersion
$ / task — Budget — Cost of stable success
10. A static set becomes stale quickly
Regression needs a holdout; reality needs fresh tasks
Static holdout
Stable regression gate
Agent-version comparison
Solutions remain hidden
Rolling live set
Fresh PRs and incidents
New data questions
Changing work environment
11. 02. Scenario families
From a business brief to incidents and architecture decisions
12. Judges change with the task
Requirements → code → review → tests → operations
13. Downstream work tests requirements
A PRD must survive implementation
Traceability from brief to AC
Functional requirements stay complete
NFRs and constraints survive
Hallucinations stay outside scope
14. Features and bugs need different oracles
The same pass rate can hide different risks
Feature coding
Spec + pre-PR commit
Hidden integration tests
Full suite and policy
Bug fixing
Issue + buggy commit
Fail-to-pass regression
Adjacent scenarios stay safe
15. Checks must find real defects
Comments and coverage alone establish very little
Code review
Severity-weighted recall
False positives are bounded
Finding accepted or executed
Test generation
Fail before, pass after
Branch and mutation coverage
Stable without flakiness
16. Incidents need a frozen environment
“What happened?” is not an eval by itself
Alert, topology, and telemetry
Runbooks and allowed actions
Triage, RCA, mitigation
Safe handoff to a human
17. 03. Open-ended domains
Architecture and data platforms have no single gold answer
18. Architecture needs more than one gold
Hard constraints → completeness → downstream executability
19. Data agents solve different problems
Analytical answers and pipelines need different evidence
20. 04. Production scorecard
Outcome, trace, operations, safety, and human acceptance
21. A scorecard replaces one overall score
Weights change by domain; safety remains a gate
22. Safety is a gate, not a penalty
Unsafe action stops release
Safety stop
Excess privilege and PII
Unsafe write or remediation
No rollback path
Human acceptance
The PR was accepted
The RCA helped on-call
The team can use it
23. One contract connects every scenario
Scenario family × input × judge × metrics × gate
Input bundle reconstructs the work
Hidden judge establishes outcome
Scorecard explains behavior
Release gate makes the decision
24. Five mistakes invalidate the result
25. Start with your own work history
Artifact → frozen state → contract → judge → release gate
26. Five decisions for production evals
What to carry into your engineering system
Evaluate replayable episodes
Check outcome and trace
Repeat the runs
Keep safety as a stop
Refresh the rolling live set
Models change. Your episode catalog remains.
27. Benchmarks establish different boundaries
Code, tools, SRE, architecture, and data platforms
SWE-bench · FEA-Bench · τ-bench
SWE-PRBench · TestExplora · AIOpsLab
R2ABench · ArchBench · DAB · BLADE
ELT-Bench · DataGovBench · coSTAR
28. Thank you!
Production-grade agent evals
The full longread, sources, and future reviews are available at polomodov.tech and in Knizhny kub
Alexander Polomodov, Technical Director & Fellow, T-Technologies
@book_cube