Diogo Almeida: AI That Does Not Need Constant Supervision

Why can AI impress us with solutions to difficult problems, while an ordinary customer refund still feels risky to delegate without checking? In an 18-minute July AI Engineer talk, Diogo Almeida suggests looking at the training objective. In his view, the industry has learned to build excellent assistants, while dependable automation requires a different set of properties. Almeida is one of the primary authors of InstructGPT, the 2022 work that helped turn a language model into a system that follows instructions. He is now examining that approach's limitations. It makes for an interesting research loop: first teach AI to work with people, then identify what prevents it from operating without constant supervision.

RLHF trains a model to receive favorable human evaluations. In the classic setup, people compare answers, those preferences train a reward model, and the language model's behavior is then optimized for its scores. Correctness can be part of the criteria, but the score remains a proxy for what we want the system to do.

Imagine a support response. It is polite, detailed, and convincing. Yet the refund has been processed under the wrong rule. A person reviewing the outcome might still catch the mistake. If the answer becomes a condition in a program, the money is already moving. The quality of the wording and the quality of the decision need separate measurements.

This leads to Almeida's provocative claim: ChatGPT and Claude Code belong to the same era—the era of assistants. In his account, an agent performing more actions does not establish that it can be trusted with an entire process. Ideally, automation would become as unremarkable as a background job on a server.

There is concrete evidence behind this criticism. The GPT-4 report shows that post-training made the model worse at estimating the probability that its answers were correct on a subset of MMLU. At the same time, its TruthfulQA results improved. A model can answer correctly more often while its confidence becomes less well calibrated. These are two distinct properties that are easy to collapse into the word “quality.”

For automation, Almeida proposes a separate objective: calibrated decisions. If a system assigns a probability of 90%, roughly 90% of a large group of those predictions should be correct. That would let developers set boundaries for when software acts on its own, requests more information, or hands a case to a person. Those boundaries still need testing on your own tasks because your decision distribution may differ.

The strong part of this argument is the demand to make uncertainty usable in software. But RLHF does not automatically imply that a person must supervise every invocation. Humans participate in training; the amount of review needed in operation depends on actual errors and their consequences. Attributing every automation problem to one training method would be too convenient.

Almeida also makes a useful point about software. AI already helps people write code faster, and he wants to expand what the resulting program can do: add semantic judgments that can be tested and combined with ordinary logic. The condition “the customer is actually asking for a refund” is much harder to express in code than a comparison of amounts or dates.

At the time of the talk, his company TypeSafe was still preparing for a release that arrived on September 15. The idea now has an API and a model called Jev, which we will discuss in the next post.

#AI #AI4SDLC #Engineering #Automation #Research