Evaluate the system, not an isolated model
The need for evals grows with accountability. When an expert reviews every assistant suggestion, errors can still be caught manually. An autonomous agent serves users who may not recognize unsafe advice, a harmful code change, or a bad translation. The cost is especially visible in a health product, where an answer must remain useful, legally defensible, and credible, while localization multiplies the combinations of languages and models that must behave consistently.
The evaluation unit should therefore be a replayable episode rather than a response or a model name. It freezes the start state, inputs, permissions, budget, hidden judge, action trace, and release criterion. Outcome says whether the task succeeded; the trace reveals unnecessary calls and constraint violations; operational measures add cost, latency, and variance across repeated runs. Together these elements form a testing discipline for a nondeterministic production system.
Four signal sources and two connected loops
Evgeny separates four practical kinds of evaluation. Deterministic code catches formal violations and computable invariants. LLM-as-a-judge handles qualities that resist a formula, but its reasoning must be retained and calibrated. Subject-matter experts label examples and turn professional judgment into reusable data. Production signals—feedback, user journeys, and technical traces—show what happens after release. A trusted expert and fewer than one hundred cases can be enough for the first useful dataset.
The offline loop compares models, prompts, context, and tools against a fixed set. The production loop collects OpenTelemetry traces, product analytics, and sampled human review, discovers missing edge cases, and feeds them back into the dataset. At Flow Health, MLflow and Databricks keep the agent’s technical path beside the customer journey. Once prompts are recognized as business logic, this connection becomes their regression-testing and continuous-improvement mechanism.
Maturity follows risk and scale
A prototype may begin with visual inspection, then add a labeled pass/fail set and shadow operation beside a human. The next stage moves evals into development and release; a mature product closes the flywheel as production traces refresh the working set, judgments supply feedback, and prompts or routing improve automatically. Platforms such as Langfuse, Braintrust, and MLflow already combine prompt, dataset, labeling, and observability workflows, so building a proprietary platform needs a separate justification.
The system remains necessary after a successful launch. A new model can improve average quality while degrading a critical scenario; a provider can decommission the current version; an old skill or instruction can turn from support into a constraint. Only an owned episode catalog can replay customer journeys, compare the replacement, and remove obsolete scaffolding safely. Trust in a model brand is not evidence, and a compelling demo is not evidence that a scalable product is close.
What to take away
- 01Evals assess the model, prompt, context, tools, permissions, and environment as one operating system.
- 02An agent trace matters as much as the answer because it exposes unsafe actions, wasted steps, cost, and required human steering.
- 03Teams can start with a small dataset, a trusted domain expert, and deterministic checks, then expand with risk and traffic.
- 04An owned episode catalog turns model, prompt, skill, or harness replacement into an engineering migration instead of guesswork.