Slide contents

1. 1. Which effort level should an agent buy?

   What high reasoning effort loses and when the lowest level is enough

2. 2. Ladder: no worse than top effort

   Terminal‑Bench 2.1: start at the lowest level, retry at the highest after a verified failure

3. 3. Three findings and one rule

   Eight models, four benchmarks, one question: which level to buy

4. 4. One setting already changed results

   Episode #46's companion: MLEvolve at medium instead of maximum effort

5. 5. Effort is bought blind

   A credence good: you see the outcome, not whether you needed the purchase

6. 6. One scale, different sets of levels

   The set of levels depends on the model; tasks, prompts and harness stay fixed

7. 7. Compare solved sets, not percentages

   Inkling on GPQA, low → medium: a net +16 hides five losses

8. 8. Even the average isn't monotonic

   Terminal‑Bench 2.1, 89 tasks: share solved by effort level, %

9. 9. Higher effort loses solved tasks

   GPT‑5.6‑sol on Terminal‑Bench, high → xhigh: +5 gained, −8 lost

10. 10. Four runs don't remove reversals

    DeepSWE v1.1: 113 tasks, four rollouts at every effort level

11. 11. Easy tasks cost several times more

    Cost of tasks solved at every level: xhigh relative to low

12. 12. Both calls and trajectories grow

    GPT‑5.6‑sol on SWE‑Bench Pro: $0.27 → $1.24 per task, 14.7 → 28.4 calls

13. 13. No reasoning doesn't mean cheaper

    GPT‑5.6‑luna on tasks solved at every level: none versus low

14. 14. Higher effort looks answers up

    GPT‑5.6‑sol on SWE‑Bench Pro, 44 tasks: original and corrected environment

15. 15. In the corrected environment, all eight drop

    Each model's top level, the same 44 tasks in two environments

16. 16. More checking, same outcome

    Claude Sonnet 5 on Terminal‑Bench: mean verification actions per task

17. 17. Three ways to pick a level

    The oracle is a bound, not a policy; self-routing and the ladder are deployable

18. 18. The ladder isn't cheaper for all

    Terminal‑Bench 2.1: the ladder's API bill relative to fixed top effort

19. 19. It's more than a second try

    DeepSWE v1.1, 113 tasks: same-level retries versus the ladder

20. 20. Waiting costs more than tokens

    Full cost = API bill + generation time at the $33.54 mean hourly wage

21. 21. A counterexample: xHigh paid off

    Claude Opus 4.7 in Claude Code: one application, 18 runs per level

22. 22. Effort is not the only lever

    The “cheap first, more on a signal” principle for turns and models

23. 23. One run per task limits conclusions

    Four limitations from the paper's text, plus one note from the host

24. 24. Start at the lowest reasoning level

    A working order for effort levels: the host's view of the paper