[1/2] Hard Won Lessons from Building Effective AI Coding Agents (Agents column)
Interesting. report from Nik Pash, Head of AI Cline The main idea is to stop complicating or why smart scaffolding kills AI agents. Before Cline, Nick worked as an engineer at Meta Reality Labs. (2019-2021), Yandex (Image Search), Samsung Electronics. The main points of the report are as follows:
1The bitter truth is that scaffolding is outdated For years, developers have compensated for the weakness of smart scaffolding models: RAG indexing, search trees, cunning tool calling systems. The problem is that now frontier models are winning without these abstractions. The ability to beat scaffolding. Nick gives the example of Gemini. 3.0which came out recently and immediately led Terminal-Bench leaderboard result 54.2% without any agency binding (If you look now, then the leader of the board is still agentic + model combinations.). By the way, Terminal-Bench Unopinionated generic stripped down harness. There is no graph search, no RAG, no indexing, just a terminal and no problem.
2️⃣ Context engineering tricks — played out Nick candidly says that instead of individual tricks for context, there is now a standard playbook to support each new model. (Sonnet 4 → 4.5, Gemini 2.5 → 3.0, GPT-5 → 5.1). Tweaks are trivial, winnings marginal. According to Nick, this topic is exhausted. There's no novelty left in it.
3The real bottleneck is a benchmark and environment for RL (reinforcement learning) In fact, the main idea of the report is buried here: you can build a cleanest agent in the world, but this will not improve the ability of the model even in the future. 1%. Nick says
Models only get better when labs train on something hard. And benchmarks, not agent cleverness... determine what frontier models learn to do next. In his opinion, the models didn’t “suddenly get better” at using tools – they got better because RL environments were built that forced them to practice specific actions: processing failure patterns, replays, error handling. Every leap in reasoning came from benchmark. Every jump in agent reliability comes from the RL environment.
Next, Nick talks about how to turn real-world tasks into training data. Cline built a system called "RL Environments Factory" - a pipeline to automatically turn real coding tasks into RL environments to train models. It looks like this.
Phase 1Qualification – Filtration of tasks Sub-agents work in parallel to see if the task is suitable for turning into an RL environment. Origins: Is there a repository? Is starting commit available? Open source? Journey: What was the user really trying to solve? What was the essence of the task? Outcome: Can we find commits/PRs that solve the problem in real life? Reject tasks of the type: slop, trivial tasks, tasks without reliable start/end states[4]
Phase 2: Building RL Environment Archaeology: Reconstructing both states (before/after) locally Documentation: document all obstacles and dependencies Containerization: Pack in Docker, remove Git (So agents can't reward hacks.) Verifier: Determine how to check the results
It is interesting that the same approach was used by the guys from the whitepaper Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks. told earlier.
4Everyone makes RL environments but nobody shares them And then Nick talks openly about the fact that every major agent lab collects this data, that is, everyone does some version of the RL environment building behind the scenes. But nobody talks about it. These companies refer to internal benchmarks, but you’ll never be able to study them because they don’t publish them openly. This data is so valuable that no one shuffles it. Agent labs stand between real-life engineers working on real-world tasks and models – they have a unique role in history.
In the sequel, I will tell you what the guys at Cline offer to improve the situation.
#AI #ML #Agents #Software #Engineering #Architecture