Understanding Foundation Models

The second episode explores foundation models: their data, domain knowledge, multimodality, and the evolution from RNNs to transformers.

The hosts unpack attention, context windows, MLP blocks, resource constraints, and model-training stages.

The final part covers SFT, RLHF, sampling strategies, why hallucinations occur, and how error costs vary by use case.