Understanding Foundation Models
What we discussed on the recording
The second episode moves from data to foundation-model behavior. A language's representation in the corpus affects quality, while domain knowledge may not fit a general model. Systems such as Med-PaLM expose trade-offs among breadth, accuracy, and cost; multimodal models add images and other signals to text.
The architecture story moves from RNNs and sequence-to-sequence models to the transformer. The hosts assemble embeddings, positional encoding, attention, and MLP blocks while explaining parameters and context windows. Attention Is All You Need and parallel sequence processing explain why the design could scale.
Compute, memory, time, and money constrain model size. SFT and RLHF follow pre-training: examples teach instruction following, while human preferences and pairwise comparisons form a reward model. Alignment is a separate engineering system rather than a small adjustment to finished weights.
A model selects its next token from a distribution, so temperature, top-k, and top-p alter repeatability and diversity. Hallucinations predate transformers, but their exact causes are not fully understood. Strategy depends on error cost: facts need constraints and checks, while stories or images may benefit from creative variation.