YC Paper Club: What Turns a Model into a Working Agent (Category #AI4SDLC)
I watched an interesting YC Paper Club session about the harness titled “Why The Harness Matters More Than The Model.” It focused on the model’s surrounding system: tools, memory, code execution, and agent workflow management. The recording appeared on September 7. Its title is debatable, but the session contains four engineering talks about what changes when the same model receives more ways to work with its environment.
1️⃣ François Chaubard’s introduction from Y Combinator works very well as an opening map. He traces the path from text generation to tools, memory, skills, subagents, and systems that modify their own harness based on their work—there are plenty of references to whitepapers here. He also demonstrates an experiment in research automation: specify an idea and a metric, then agents search for papers, run experiments, and prepare a manuscript. The roles even include a research supervisor who periodically reminds the researcher that it is time to move forward. Academic life is getting automated too :)
2️⃣ Seth Karten from Prime Intellect discusses Prime Agent. He uses an interesting analogy with a computer’s memory hierarchy: model weights, active context, variables in a running Python process, and persistent files. A large tool result can remain in process memory, be processed with code, and contribute only the necessary fragment to the model’s context. A subagent can return to a task with its context preserved. Skills and instructions can be updated as work proceeds so useful experience survives an individual session. By the way, Karten’s idea of “letting agents talk to each other” grew out of a very human problem: he was tired of manually carrying information between assistants working in parallel.
Examples include building game-system emulators, optimizing GPU kernels, and running multi-day research tasks. There is also a story about an almost perfect ARC-AGI result. According to Karten, the first run scored 99.9%, but inspecting the logs revealed cheating. They had to fix the environment isolation and run it again. That is a useful detail for anyone accustomed to looking only at a benchmark’s final number.
3️⃣ Jon Saad-Falcon from Stanford presents OpenJarvis, a personal assistant that runs on the user’s own device. The configuration loop is interesting: a strong cloud model studies the local system’s failures and proposes changes to its model, tools, memory, and agent logic. The changes are validated, while the resulting configuration handles tasks locally. The authors report roughly an 800-fold reduction in marginal API cost in their tests. That is not total cost of ownership once hardware and electricity are included. Still, the possibility of using a cloud model to prepare a cheaper local assistant deserves attention.
4️⃣ The most applied section is Josh France and Regan Bell from YC Labs describing QM, an internal agent platform for YC employees. Before QM, the team had launched more than 50 Hermes agents in virtual machines. The assistants were useful but needed configuration and constant maintenance: someone had to enter individual instances and fix things.
QM moved conversation history and state into PostgreSQL, while sandboxes became resources that an agent attaches when needed. They also discuss less obvious difficulties: an agent finishes too early; automated fixes help one area without accounting for the whole system; information from a private conversation can surface in a public channel because the model poorly understands social context.
Overall, I recommend watching the original recording. It is only an hour long and contains both research ideas and practical experience.
#AI4SDLC #AI #Agents #Engineering #PlatformEngineering