Jediah Katz on the harness: what remains after a new model (#AI4SDLC)
Jediah Katz, an engineer from Cursor now at SpaceXAI, explores an interesting process in an 18-minute talk: a team improves its harness, and the next model lets it delete some of that work. The recording was released on AI Engineer on October 19 under the title “Smarter Models Killed the Harness. Here's What's Left”.
According to Katz, Cursor once had to give the model explicit instructions on exploring the repository and checking the result, and use an auxiliary model to apply edits. Gradually, models learned these actions. His explanation is that the harness makes harder tasks accessible, useful trajectories emerge, models learn from them, and some of the old supports become redundant. This is an engineer's view from inside a product, rather than independent proof that complex harnesses are no longer needed anywhere.
The process continues. In Grokbot, where a persistent coordinating agent distributes work among executors, the team is adding instructions and interventions again. Now the system needs to learn its way around websites, provide timely progress updates, understand when to act and when to ask for clarification, and remember the person's preferences. On a new class of tasks, the model needs help again, but this is starting to look like a meta-harness:)
Katz separately identifies these enduring parts of the harness: - Execution: where the agent lives, how state is preserved, and how work resumes after a failure. - Authorization: which data and actions it can access, and where its budget ends. - Efficiency: the cost of context, repeated calls, and model choices. An identical benchmark score does not mean an identical cost per task.
The juiciest detail is the bookkeeping for temporary supports, which the team calls shims. For example, the agent may need a reminder that the user is waiting for a response. According to Katz, these interventions are tracked centrally, their activations are logged, and a bot proposes PRs to remove those that have stopped firing.
The talk closely echoes what I discussed at Giga Conf on October 16: accumulated instructions deserve a review after a new model arrives. Otherwise, the harness gradually becomes a museum of earlier generations' limitations. In Research Insights #38, I examined how to test the contribution of individual components: which failure does a plan, context management, or a tool fix? And in the longread on harness evolution, I proposed an assumptions register: why a component was added, which model it helped, what evidence there is, and when to reassess it.
Katz shows how to connect that register to observations of real work. But “it stopped firing” is only a reason to run an experiment. The relevant tasks may have disappeared, logging may have broken, or a rare failure may simply not have recurred for a long time. Here is how I would test removing a component:
- Keep the episodes that led to the support being added, and add representative current tasks. - With the same model version and the same budget, compare the full harness with a version lacking that component (this is an ablation study). Repeat the runs - Define the acceptable degradation in advance. Look at quality, cost, time, and human interventions; if there is too little data, leave the question open. - Attach the results to the PR and retain the option to restore the component if a regression occurs.
Authorization boundaries are a separate matter. A prohibition on deleting other people's data cannot be lifted just because the agent has not violated it for a long time. Greater model capability does not, by itself, expand its permissions. And in the next PR that adds another rule for an agent, I would immediately ask for this to be recorded: what test result would allow us to remove that rule?
#AI4SDLC #Agents #Engineering #Architecture #Evals