Building Evals That Work — with Evgeny Sergeev

Alexander Polomodov and Evgeny Sergeev discuss the shift from a successful answer to a repeatable quality system. Evals matter more as agents gain autonomy and users become less able to spot unsafe advice or harmful changes. In a health product, an answer must be useful, legally defensible, and credible at once.

The evaluation unit is a replayable episode, not a response or model brand. It freezes start state, input, permissions, budget, hidden judge, trace, and release criterion. Outcome records success, the trace explains the path and violations, and measures add cost, latency, and variance.

Signals come from deterministic code, a calibrated LLM judge, experts, and production feedback. Offline evals compare models, prompts, context, and tools on a fixed set; production returns edge cases to the dataset. A trusted expert and fewer than one hundred cases can be enough for a first useful suite.

Maturity moves from manual review to labeled pass/fail, shadow mode, release gates, and production traces. The loop stays necessary because a new model may break a critical path, a provider may retire a version, and an old skill may constrain performance. An owned episode catalog enables safe comparison and removal; a polished demo does not.