Skip to content
back to the archive page
#AI

Stanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 4 – Reinforcement Learning with LLMs (Series #AI)

I finally finished the fourth lecture in Transformers & LLMs, on reinforcement learning. It’s the most intense lecture in the course so far. In my breakdown of lecture three, an agent took nineteen useful actions and broke the program on the twentieth. The tests failed at the end. How do you tell the model which of those twenty steps to change? In lecture four of Stanford CME295, Afshine and Shervine Amidi explain how that feedback becomes an update to the model’s weights.

First, who gives the score? For tone or clarity, a person can pick the better of two answers. Those preferences train a reward model that scores new answers. That’s RLHF (Reinforcement Learning from Human Feedback). For tasks with verifiable outcomes there’s RLVR (Reinforcement Learning with Verifiable Rewards): check a number against a reference or run tests on generated code.

At each step the model picks the next token given the prompt and the answer so far. In REINFORCE, a classic policy gradient method in reinforcement learning, the model generates an answer and the reward weights the gradient of its log probability. That’s how a final score affects the probabilities of individual tokens. Single scores make for noisy updates. Each new answer also has to be generated a token at a time. For an existing answer you can recompute token probabilities in one pass, so it pays to reuse a batch for several updates.

🔸 PPO estimates what to expect at each step A critic trained alongside the model looks at the partial answer and predicts the reward for continuing. That prediction and what actually happens later give an estimate of the advantage, or how good the choice turned out to be. Clipping limits the incentive to change probabilities too far while reusing a batch. The lecture setup also has a KL penalty for straying from the original model. Clipping takes the version that just generated the batch as its reference. The KL penalty uses a fixed starting model.

🔸 GRPO gets its reference from the attempts themselves Suppose the model proposed four fixes and only one passed the tests. It scores above the group average and the others below. The comparison needs no separate critic. If all four fail with the same score, all four get zero advantage. With outcome-only scoring, every token in an answer gets the same advantage. Even a successful attempt may include several entirely unnecessary steps that get the same signal.

In the original GRPO a token’s contribution also depends on answer length. The longer the answer, the less each token weighs in the average. DAPO averages across all tokens in the group, and Dr. GRPO removes the division by the length of an individual answer.

🔸 On-policy distillation helps wherever the student actually ends up The student starts solving a task. The teacher sees what the student has written so far and shows which continuations it considers likely. The lecture expresses this through reverse KL: if the student often picks a token the teacher considers very unlikely, the divergence costs more. In self-distillation the same model acts as teacher, with additional information such as a reference solution during training. You can learn from your own dead ends if the teacher can show you a way out.

All in all, an excellent lecture. It took me a week to watch and about three hours of screen time in total :)

#AI #LLM #Learning #Engineering #Agents

Open video on YouTube