[1/2] Stanford CME295: How LLMs Are Trained (#AI)
In Stanford CME295’s third lecture, Afshine and Shervine Amidi explain how a model that predicts the next token becomes a useful assistant. What interests me here is the connection between training methods: each gives the model a different signal about how it should respond.
First comes a general foundation. During pretraining, the model learns to continue text, code, and mathematical solutions. Cross-entropy measures how well it predicts tokens from its training data: the higher the probability of the correct next token, the lower the loss. That makes data composition crucial: someone has to decide what to filter, how to mix sources, and which material to repeat.
During midtraining, training continues on more targeted data; this is also where the context window may be extended. Simply allowing the model to accept more tokens is not enough. It needs examples where continuing the text actually requires information from much earlier in the sequence.
From here, it helps to compare methods along two questions: whose responses does the model learn from, and how detailed is the feedback?
🔸 SFT (supervised fine tuning) — demonstrate the desired behavior We provide pairs of prompts and good responses. The model still predicts the next token, but in the setup shown, the loss is computed over the response, with the prompt serving as a condition. This can teach instruction following, dialogue formats, and the desired style. The authors also discuss LoRA (low-rank adaptation): the base weights are frozen, and their update is represented by the product of two small trainable matrices. This is a way to reduce the cost of adaptation; it is not a separate mandatory stage after SFT.
🔸 Preference tuning — compare responses People often find it easier to choose the better of two answers than to write an ideal one. In the RLHF setup described, these preferences first train a reward model. It then evaluates new responses from the LLM, and training increases the expected score while limiting divergence from the reference model. The evaluator can be wrong, so a high score does not yet guarantee quality.
DPO allows training directly on preference pairs without a separate reward model. In its basic form, the pairs are fixed: fresh attempts from the current model are not required at every step.
🔸 RL (reinforcement learning) — evaluate the outcome of an attempt If the reward is verifiable, a mathematical answer can be checked, and a program can be tested. The lecture explains this through GRPO: the model generates several solutions and learns from their relative success. GRPO itself is a training algorithm; verifiability comes from the chosen evaluation method. But a score for the whole solution says little about exactly where the mistake happened. An agent might take nineteen useful actions and break the program on the twentieth.
🔸 On-policy distillation — a teacher helps on the student’s own attempts The student generates a continuation, and the teacher provides a signal about next-token probabilities in those same contexts. The student is trained to reduce divergence from the teacher. The feedback is detailed, although its quality still depends on the teacher.
This brings us back to a student’s question at the start of the lecture. With teacher forcing, the model receives the correct preceding tokens from the dataset, even if it would have predicted something else. In use, it continues its own text, including its mistakes. Learning from its own attempts makes it possible to work with situations the model actually encounters.
In part two, I will cover the practical example of GLM-5 that the authors discussed in the lecture. They used it to show how all these theoretical approaches are combined in practice, and what to do if a new training stage degrades skills the model has already acquired.
#AI #LLM #Learning #Engineering #Agents