Turning AI-Agent Evals into an Engineering System

The need for evals grows with accountability. When an expert reviews every assistant suggestion, errors can still be caught manually. An autonomous agent serves users who may not recognize unsafe advice, a harmful code change, or a bad translation. The cost is especially visible in a health product, where an answer must remain useful, legally defensible, and credible, while localization multiplies the combinations of languages and models that must behave consistently. The farther people move from each decision, the more verification must live inside the product. One quality score cannot cover every kind of harm, so criteria separate mandatory constraints, usefulness, and cases requiring expert escalation.

The evaluation unit should therefore be a replayable episode rather than a response or a model name. It freezes the start state, inputs, permissions, budget, hidden judge, action trace, and release criterion. Outcome says whether the task succeeded; the trace reveals unnecessary calls and constraint violations; operational measures add cost, latency, and variance across repeated runs. Together these elements form a testing discipline for a nondeterministic production system. Replayability does not require identical wording; it requires identical conditions and comparable acceptance. Repeated runs matter because one successful outcome reveals neither the behavior distribution nor rare dangerous paths.

Evgeny separates four practical kinds of evaluation. Deterministic code catches formal violations and computable invariants. LLM-as-a-judge handles qualities that resist a formula, but its reasoning must be retained and calibrated. Subject-matter experts label examples and turn professional judgment into reusable data. Production signals—feedback, user journeys, and technical traces—show what happens after release. A trusted expert and fewer than one hundred cases can be enough for the first useful dataset. The signals do not substitute for one another: a validator cannot judge usefulness, a model judge may fail systematically, and manual labeling is too expensive for every run. Their combination directs scarce attention toward ambiguity after cheap checks remove obvious failures.

The offline loop compares models, prompts, context, and tools against a fixed set. The production loop collects OpenTelemetry traces, product analytics, and sampled human review, discovers missing edge cases, and feeds them back into the dataset. At Flo Health, MLflow and Databricks keep the agent’s technical path beside the customer journey. Once prompts are recognized as business logic, this connection becomes their regression-testing and continuous-improvement mechanism. A fixed holdout must remain unchanged, or the system only improves on yesterday's failures and loses an honest version comparison. New production cases enrich the working set while the release threshold remains predefined.

A prototype may begin with visual inspection, then add a labeled pass/fail set and shadow operation beside a human. The next stage moves evals into development and release; a mature product closes the flywheel as production traces refresh the working set, judgments supply feedback, and prompts or routing improve automatically. Platforms such as Langfuse, Braintrust, and MLflow already combine prompt, dataset, labeling, and observability workflows, so building a proprietary platform needs a separate justification. Buying a platform cannot skip these stages. If a team has no shared accepted outcome or consistent labeling practice, a sophisticated interface merely stores contradictory judgments neatly and creates an illusion of maturity.

The system remains necessary after a successful launch. A new model can improve average quality while degrading a critical scenario; a provider can decommission the current version; an old skill or instruction can turn from support into a constraint. Only an owned episode catalog can replay customer journeys, compare the replacement, and remove obsolete scaffolding safely. Trust in a model brand is not evidence, and a compelling demo is not evidence that a scalable product is close. A regression can hide behind a better average, so critical slices require separate checks by language, market, and failure cost. Release authority belongs to the organization's criterion, not a provider leaderboard.