Harness at Giga Conf: Talk Materials

I've put together the materials from my October 16 talk at Giga Conf, “Harness: The Past, Present, and Future of the Layer Between the Model and the World.” There's a recording, a slide deck, and a long read on the topic. I've talked a lot about tools, memory, context, and checks. Here I put them together around one question: which of the things we've built will we still need after the next model comes out?

The metaphor I use in the talk is words & walls. Some instructions compensate for the weaknesses of a particular model: how to maintain a plan, when to reread a file, and how to check its own work. Their usefulness is worth testing again after a model upgrade. Permissions, isolation, and restrictions on actions are boundaries that code must enforce regardless of the model. Being good at a task doesn't in itself grant permission to carry it out.

In the talk I cover:

🔸 How the harness evolved
From a loop in the prompt to full-fledged agent products and managed execution. Which functions are moving into the model and API, and which remain in the environment?
🔸 What we're actually choosing
An all-in-one tool, a standalone agent, or an SDK with which to build our own system. I also look at why similar tools can have very different defaults for access and execution.
🔸 How to demonstrate a component's value
Remove it in a separate experiment and compare the results on your own tasks. A registry helps: what assumption did we make about the model, how do we test it, and when should we revisit it?
🔸 What to measure
How consistently the system succeeds and what an accepted result costs once you include review and rework. One successful run tells you little about how an agent will work day to day.
🔸 Where this is heading
My hypotheses about agents with their own machines, memory, and identity, and about execution and action checks shifting to the vendor. Each bet comes with a condition that would make me revisit it.

One important qualification: it's the compensating instructions that are worth revisiting. Task requirements, completion criteria, and project knowledge still matter when the model changes.

P.S.
I already wrote about this talk and promised to make an extended episode if the post got 100 🔥. It's only at 70 so far, but it's in your hands.

#AI4SDLC #Agents #Engineering #Security #Conference