Skip to content
#AI

[1/2] John Schulman on dead ends, scaling RL, and building research institutions (AI column)

#AI #Engineering #Metrics #Software #Architecture #RnD #ML

I saw it. Another episode of Cafe Cursor podcastMichael Truell and John Schulman. Michael is the co-founder of Cursor, and John is the co-founder of OpenAI and the man behind RLHF. (reinforcement learning from human feedback)The technology behind ChatGPT. He is currently in Thinking Machines as Chief Scientist. In this podcast, John shares insights into how things worked in OpenAI. 2016 What went wrong and where does reinforcement learning go? Highlights of the interview below

1Speedrunning ChatGPT: Could you do it? 2018 Shulman believes that with full hindsight ChatGPT-3.5 could have been assembled in 2018–2019 forcefully 2–3 A man on a pair of GPU boxes. The secret is not compute scale, but smart post-training: the right fine-tuning dataset compensates for the smaller model. Compares it to nanoGPT (One person, one boxing, six months). Conclusion: Scaling is important, but smart trick> brute force.

2Early OpenAI In 2016–2017 OpenAI was like an academic group of people working on 1–3 Man over whitepapers to his liking. For example, there was an interesting project Universe as an attempt to create a universal RL agent on hundreds of video games and web tasks. The idea was right, but 10 The years were ahead of time - then the models were not generalized, the system was clumsy. Later came results from Dota and Procgen. (gameplay). The direction of robots then was also deadlocked, but useful - it pumped the team on large engineering projects and trained people in system work.

3Why is value functions not in fashion? RLHF and verifiable rewards (even on 10k+ tokens) value (value functions) They don't reduce variability. Shulman is waiting for their comeback, but so far Policy Gradient methods are winning on short horizons. [1]

4️⃣ Continual learning: long context + LoRA According to John, two factors are needed for continuous learning.

  1. In-context learning (long context) - for fast, short-horizon.
  2. Parameter fine-tuning (LoRA) For long-horizon knowledge that requires capacity. Scaling can solve the problem without new ideas, but Shulman expects breakthroughs that shift scaling laws.

5Brittle generalization: people vs models Cool models in-context (human)But worse on long horizons - stuck where a person is capable of self-correction. Why? Human beings have evolved to 80- summer timeline. It is unclear if this is a temporary or fundamental limitation. Testing for decades is equivalent to running evals for decades, which is problematic.

6The future of reinforcement learning: GANs 2.0 multi-agent games Shulman is waiting for the return of ideas from 2010-x:

  • Co-training generators + verifiers (like GANS) This is about a self-reinforcing cycle: better reasoning → better verifier → better generator. [1]
  • Multi-agent games** (zero-sum/debate) Automatic curriculum + theoretical guarantees from complexity theory (Polynomial judge creates incentives for complex problems (conditionally)). OpenAI/Anthropic's Debate Game is an underappreciated idea.

In continuation I'm going to talk about other interesting theses, like approaches to research leadership or slowing down breakthroughs.

#Engineering #AI #Metrics #Software #Architecture #RnD #ML