Stanford CME295, Lecture 2: What Changed Inside the Transformer (#AI)
The recording of the course’s second lecture on transformers and LLMs is now available. In the first lecture breakdown, we went from tokenization to the Transformer described in the 2017 paper. In the second lecture, Afshine and Shervine Amidi show how that structure develops into modern LLMs. This is a good point to test yourself: how well do you remember the reasons behind the familiar acronyms?
Over 1 hour and 43 minutes, they cover the BERT and GPT families, the balance between model size and training data, architectural changes, and generation strategies. A few points stand out.
🔸 MoE: a large model does not have to use every weight for every token.
Several “experts” replace a single feed-forward block, and a router chooses which ones to activate. That choice is made for each token. But an interesting problem follows: experts chosen early in training receive more updates, improve, and get chosen even more often. The others sit idle. This is routing collapse; the lecture explains how an auxiliary penalty for imbalance helps distribute the load. It is a good example of how much work sits behind a simple diagram with a router.
🔸 RoPE: why rotate vectors in the first place?
The explanation is particularly good here. It starts with positional representations illustrated through clock hands moving at different frequencies, then moves on to rotating queries and keys. Their dot product gains a dependency on the difference between the tokens’ positions. This makes it clear how token order enters attention directly. Knowing what RoPE (Rotary Position Embedding) stands for and understanding where the rotation comes from are quite different levels of familiarity.
🔸 Attention has to fit in memory.
During generation, keys and values for previous tokens can be stored and reused. But longer sequences require more memory. Hence GQA (Grouped-Query Attention): several query heads share keys and values. Another technique is sliding window attention, where each layer only attends to nearby tokens. Across several layers, however, information can travel beyond a single window. The instructors make a useful connection to receptive fields in convolutional networks.
🔸 Even temperature = 0 does not promise identical answers.
After greedy decoding, beam search, top-k, and top-p, the lecture covers a detail that is easy to miss. The server batches requests together, and a change in batch size can alter the order of computations. Floating-point addition depends on operation order; small numerical differences can affect token selection. Reproducibility therefore also depends on the inference implementation. The Thinking Machines article referenced in the lecture explains the details.
🔸 Valid JSON can be enforced during token selection.
Guided decoding restricts possible continuations according to the required syntax. At each step, the model chooses among the allowed tokens. This explains what makes a format guarantee possible; the correctness of the data inside the JSON still needs to be checked.
The closing section briefly covers the context window, context rot, learning from examples in a prompt, and reasoning techniques. A useful reminder: being able to fit a long text does not guarantee that a model will reliably use every part of it.
Even if you already know how an LLM works, these connections make the recording worth watching. Afterwards, try opening an architecture diagram for a familiar model and explaining why it has MoE, RoPE, or GQA, where it saves computation, where it saves memory, and what determines the answer it produces. The RoPE explanation alone deserves a pause with a pencil.
P.S. I have read papers and various books on this subject many times, but the presentation here is really good: it helps refresh the fundamentals and organize them more clearly in my head.
#AI #LLM #Learning #Engineering #Architecture