PRD == evals: How AI Brings Product and ML Closer

In classical ML, an engineer decomposed a business problem, chose a metric, and built a dataset. A conventional PRD described nondeterministic outcomes poorly, so product, analytics, and ML shared responsibility without one quality owner. Model APIs and coding agents changed the balance: a product manager can build a prototype, while an engineer embeds an existing model into a business process. Their roles converge where acceptable behavior is defined.

PRD == evals does not mean replacing a document with a spreadsheet. An eval captures scenarios, good-answer criteria, prohibitions, and regressions, making it a product control surface. For an investment RAG assistant, the team samples production request-response pairs, clusters failures, then tackles the largest class: stale data, a missing source, a wrong number, an unsuitable tone, or a request that belongs elsewhere. Repaired cases remain as regressions while new logs refresh the suite. A perfect score on a static set suggests overfitting, not flawless quality.

An eval set requires product and domain judgment. Annotators disagree about classes and scales, so teams need a shared rubric, calibration, and binary criteria where possible. Conventional tests check structure, LLM-as-a-judge assesses meaning, general annotators judge everyday appropriateness, and expensive domain specialists interpret regulations and difficult answers. Run frequency reflects the cost of people, inference, and failure.

Offline evaluation needs runtime constraints. An investment assistant must not give personalized advice, so the team used a disclaimer on every answer plus guardrails. A children's assistant needs different filters and post-generation checks; each layer reduces risk but increases latency. Munirova recalls building a RAG assistant in 2023, before mature techniques such as RAGAS, and learning through eval mistakes. A fast MVP therefore proves little: production needs fresh data, routing, permissions, tools, memory, security, and named quality owners.

A general assistant raises expectations beyond the system's reach. The early Oleg assistant, created before LLMs, exposed the trap: users expected help with anything, while limited data and skills produced unpredictable answers. T-Bank's later “universe of assistants” divided work by domain, selecting a broad audience, an acute problem, and bank expertise. Specialization supports distinct evals and guardrails for investors, children, or travelers, although users must choose the assistant.

An AI Product Builder moves from user need and prototype to a solution and improvement loop, owning quality and value. This need not be a new role or a one-person team: integrations and domain risks still require specialists. Yet a five- or six-person team loses less time to handoffs. Product managers should learn prototyping and eval design; ML engineers should learn user journeys, operating processes, and deployment economics. Stronger models may erase prompt tricks, but not domain judgment, fresh test sets, or infrastructure with explicit policies, automated checks, and a safe path to production.