Why a successful AI product POC (proof of concept) is still not production-ready (Rubric #AI4SDLC)

I watched a new talk from AI Engineer World’s Fair that extends the discussion of the FDE (Forward Deployed Engineer). Last time, I wrote about the FDE as an engineer who embeds with a customer team and turns AI into a working process. The new talk shows the next act: the POC works, but production has not even begun.

Christopher Lovejoy of Anthropic and Saul Howard, VP of Engineering at Anterior, open with a hypothetical scenario. Two engineers build an agent for a medical workflow in four weeks. Accuracy is good; the system is fast and relatively inexpensive. Everyone is pleased at the demo: finance calculates the budget, the medical director is ready to tell colleagues about the accuracy, and sales already wants “Powered by AI” on the website.

The next day brings a different set of questions. Where is the complete audit trail? How does medical data move? Who approves a disputed decision? Can an untrusted document steer the model? How will quality be checked after release? The attractive POC stops there: it was optimized for the right answer, not for proving every action.

The speakers propose reversing the order: make production constraints load-bearing first, then recover the prototype’s accuracy. That requires three primitives:

1️⃣ Append-only event log — one history of actions, access, and authorization. It makes state reproducible and supports audit, but the trade-off has a cost: writing is easy, reading is harder, and projections and snapshots are required; 2️⃣ Schema-driven object storage for the medical data itself. The log holds references, access is granted at use time, and an engineer can debug the agent’s path without seeing PHI; 3️⃣ One action contract for the LLM and the human. Any step can be handed to a person in the middle of a workflow, while the shared context can be rendered either as a prompt or as UI. The interface is shared; the authority is not.

The authors argue that evals follow from this foundation: replay the same episode with a different prompt, model, or code; compare the agent’s action with a human’s; and run checks against production data inside the customer’s boundary. What I like is the order of thought. If audit, data boundaries, and human approval appear in the plan only “after a successful POC,” the architecture is already late. In a regulated system, these are not gates before release; they are the data and execution model itself.

#AI4SDLC #AI #Agents #Architecture #DevSecOps #Evals #Engineering