A harness that improves the harness
A loop that predicts each edit's effect: what it found, where the gain lives, and what it misses
Slide contents
1. A harness that improves the harness
A loop that predicts each edit's effect: what it found, where the gain lives, and what it misses
2. From the loop to numbers and limits
The model stays fixed; the prompt, tools, middleware and memory change
3. Hand-tuned harnesses lag behind new models
A harness tuned for one model often underperforms on another
4. The bottleneck is observability, not capability
Three pillars: components, experience and decisions
5. Seven component types live as files
The seed is one bash tool and three rules
6. A debugger distills trajectories into reports
Overview first, then a per-task report, raw traces as a last resort
7. Every edit ships with a prediction
What it fixes, what it may break, and why this component
8. Each round runs, checks, rolls back, edits
The best harness is picked by pass@1 on the same tasks
9. One trial is 0.56 points
89 tasks × 2 trials; selection and scoring on one set
10. Ten iterations: 69.7 → 77.0%
Above Codex, ACE and TF-GRPO; on Hard, Codex and TF-GRPO lead
11. The loop shipped words backed by walls
Four winning rounds: rules for the model, mechanisms in code
12. The rule existed; the file still vanished
path-tracing: a shell guard intercepted rm -rf after the check
13. The gain sits in memory, tools, middleware
One layer at a time on the seed; the prompt alone did not help
14. There is no “AHE minus prompt” ablation
−2.3 points is four trials, about two tasks of 89
15. Each component targets its own failure surface
Gains diverge across difficulty tiers
16. The loop predicts fixes, not regressions
Precision and recall of predictions over nine rounds
17. On SWE-bench Verified, accuracy barely moves
75.6% vs 75.2% at 461K vs 526K tokens per trial
18. All five other models gained
Other families +5.1 to +10.1; GPT-5.4 medium and xhigh +2.3 each
19. Meta-Harness: 80 lines on top of KIRA
Opus 4.6 on TB2: 76.4% vs Terminus-KIRA's 74.7%
20. Off-distribution, AHE ended below its seed
RRSI: same seed, Claude Opus 4.8, Harvey LAB tasks
21. Authors: a research prototype, not a product
Three limits from the paper and two of ours
22. Harness ledger: assumption and trigger
A ledger row works like a manifest entry, plus held-out checks
23. Self-evolution works but needs outside checks
Each harness edit is a testable prediction
TB2: 69.7 → 77.0% on the same tasks
Gains sit in tools, memory and middleware
The loop barely foresees regressions
Beyond TB2 the gain shrinks or vanishes
Let the loop edit, not accept