Tokenizer, nanoGPT and multimodal models

BPE training: start with the text as a sequence of bytes (256 base tokens, so any string is representable). Count every adjacent pair, merge the most frequent pair into a new token, replace it everywhere, and repeat until the vocabulary reaches the target size (GPT-2: about 50k; modern models: 100k–200k+).

Encoding new text applies the learned merges in order. Decoding concatenates the bytes of each token. Frequent words become single tokens; rare words split into pieces; nothing is ever out-of-vocabulary. Step through merges in the simulation.

Production tokenizers first split text with a regex (so merges don't cross word, number or punctuation boundaries) and add special tokens like <|endoftext|>.

Spelling and character counting are hard because 'strawberry' may be one or two tokens; the model never directly sees its letters. Arithmetic is awkward because numbers split inconsistently ('1234' vs '12' + '34').

Non-English languages and code often take 2–4× more tokens for the same content, so they cost more and fill context faster. Leading spaces matter: ' hello' and 'hello' are different tokens.

Trailing whitespace in prompts, unusual unicode and rare 'glitch tokens' can produce odd behaviour. When a model misbehaves on a specific string, inspect its tokens first.

nanoGPT is a compact, readable codebase that reproduces GPT-2. Compared with your from-scratch model: fused, efficient attention (scaled_dot_product_attention, i.e. FlashAttention), weight tying between the embedding and output layers, careful initialisation, AdamW with weight decay applied only to 2D weights, cosine LR schedule with warmup, gradient accumulation, mixed precision, torch.compile, and distributed data parallel training.

Reading it end to end is the bridge from 'I understand GPT' to 'I can read production training code'.

Optional: Karpathy's 'Let's reproduce GPT-2 (124M)' walks through the same engineering live.

Pretraining: next-token prediction over a huge filtered web, code and book corpus. The result is a base model that continues documents but doesn't follow instructions.

Post-training: supervised fine-tuning (SFT) on curated conversations teaches the assistant format; preference optimisation (RLHF or DPO) on human or AI preference judgements shapes helpfulness and harmlessness; reinforcement learning on verifiable tasks (maths, code) builds reasoning.

Karpathy's 'Deep Dive into LLMs' covers this whole pipeline and is the bridge into week 26.

The common recipe: a pretrained vision encoder (a ViT, often CLIP-style) turns an image into patch embeddings; a small projector maps them into the LLM's embedding space; the LLM then processes these 'visual tokens' alongside text tokens with ordinary attention.

Training typically aligns the projector first (image–caption pairs), then fine-tunes on visual instruction data. Some models are natively multimodal from pretraining.

Practical consequences: images cost tokens (often hundreds per image, scaling with resolution), small text in images may be missed at low resolution, and document understanding (Project 3) relies on exactly this.

Kaplan et al. (2020) found that language-model loss decreases as a smooth power law with model size, dataset size and compute, over many orders of magnitude. This made it possible to predict a big model's performance from small runs.

The Chinchilla paper (2022) showed earlier models were undertrained: for a fixed compute budget, parameters and training tokens should grow roughly equally, around 20 tokens per parameter. Training compute is roughly $6ND$ FLOPs for N parameters and D tokens.

In practice, models are now trained far past Chinchilla-optimal, because a smaller model trained longer is cheaper to serve. Inference cost, not just training cost, drives the choice.