LLM 101

Pretraining: next-token prediction on trillions of tokens. Produces broad knowledge and skills, but a base model that continues text rather than following instructions. It is by far the most expensive stage.

Supervised fine-tuning (SFT): train on thousands to millions of curated prompt→response conversations. The model learns the assistant persona and format. RLHF trains a reward model on human preference comparisons and optimises the LLM against it (classically with PPO). DPO skips the reward model and optimises directly on preference pairs: simpler and now common.

Newer stages use reinforcement learning with verifiable rewards (unit tests passing, maths answers matching) to train long reasoning. As an engineer you mostly consume these models, but knowing the stages explains behaviour: sycophancy, refusals and verbosity are post-training artefacts; knowledge gaps are pretraining artefacts.

Everything is metered in tokens: roughly 0.75 English words per token, more for code and non-English text. Providers charge separately for input tokens (everything you send: system prompt, history, retrieved documents, tool definitions) and output tokens (what the model generates). Output tokens are usually priced several times higher.

The context window is the maximum input + output tokens in one call. Long windows are convenient, but every token in the window is processed on every call: a 50k-token system prompt is paid for on every turn of a conversation, unless it is cached.

Cost per request = input tokens × input price + output tokens × output price. In an agent loop, the input grows each step because the history is resent, so cost grows roughly quadratically with the number of steps. Budget it explicitly.

At each step the model emits logits over the vocabulary. Temperature divides the logits before softmax: below 1 sharpens the distribution (more deterministic), above 1 flattens it (more random); 0 means greedy. Top-k keeps only the k most likely tokens; top-p (nucleus) keeps the smallest set whose probabilities sum to p.

Guidance: low temperature (0–0.3) for extraction, classification, code and anything you'll parse; moderate (0.7–1.0) for creative writing and brainstorming. Change temperature or top-p, not both at once.

Even at temperature 0, outputs are not perfectly deterministic across calls (batching and floating-point effects). Design for variation: validate outputs, don't assume byte-identical repeats. Reshape a distribution yourself in the simulation.

Prefill: all input tokens are processed in parallel in one forward pass, computing and caching each layer's keys and values. It is compute-bound and fast per token. It determines time to first token (TTFT).

Decode: each new token requires a full forward pass for just that position, attending to all cached keys and values. It is sequential and memory-bandwidth-bound. It determines tokens per second, and is why long outputs are slow and why output tokens are priced higher.

The KV cache stores keys and values for every previous token, so each decode step is incremental instead of recomputing the whole context. It grows linearly with context length, often dominating GPU memory at long contexts. Prompt caching reuses the prefill of a shared prefix across requests. Watch both phases in the simulation.

Rule of thumb: weights take parameters × bytes per parameter. A 7B model needs ~14 GB at 16-bit (bf16), ~7 GB at 8-bit, ~3.5–4 GB at 4-bit. Add the KV cache (grows with context and batch size) and runtime overhead (~10–20%).

Quantisation stores weights in fewer bits (int8, 4-bit formats like GPTQ, AWQ, GGUF's Q4_K_M) with modest quality loss, often small at 8-bit and noticeable but usually acceptable at 4-bit. That is what makes local inference with Ollama or llama.cpp possible on a laptop.

Training needs far more memory than inference: gradients and optimiser states (Adam keeps two extra values per parameter) multiply memory several-fold. That is why QLoRA (4-bit base + small trainable adapters) matters.

Reasoning models are trained to produce long internal chains of thought before answering, trading latency and tokens for accuracy on maths, code and multi-step problems. This is test-time compute: more thinking at inference can substitute for a bigger model. Use them selectively; for simple extraction they're slow and costly.

Hallucination is structural: a language model generates plausible continuations, and nothing in next-token prediction guarantees truth. Models are worse on rare facts, precise numbers, citations and anything after their training cutoff, and they are often confidently wrong.

Engineering mitigations rather than hopes: ground answers in retrieved sources (RAG), require citations, allow and reward 'I don't know', validate structured outputs, and measure with evals.

In a mixture-of-experts (MoE) transformer, the MLP in each block is replaced by many 'expert' MLPs plus a small router that sends each token to the top-k experts (often 2 of 8, or 8 of hundreds). Only the chosen experts run for that token.

So a model can have a very large total parameter count while each token uses a small active parameter count. Compute per token tracks active parameters; memory tracks total parameters, since all experts must be loaded.

Trade-offs: more memory and more complex serving (expert parallelism, load balancing across experts) in exchange for better quality per unit of compute. Many frontier and open models are MoE; check 'total vs active parameters' on model cards when estimating cost and VRAM.

Decoding is slow because each token needs a full pass of the big model. Speculative decoding lets a cheap draft (a small model, or extra prediction heads) propose several tokens; the large model checks them all in a single parallel forward pass and accepts the longest correct prefix.

With the right acceptance rule, the output distribution is exactly the large model's. When the draft is usually right (predictable text, code, structured output), speedups of 2–3× are common.

It's mostly a serving-side optimisation (vLLM, TensorRT-LLM and many hosted APIs use it), but it explains why latency can vary with how predictable the output is.

Public signals: benchmark suites (knowledge, reasoning, coding, maths), human-preference arenas, and independent price/speed trackers. They're useful for shortlisting but suffer from contamination (test data leaking into training) and may not resemble your task.

Decide with your eval set: 50–200 representative inputs scored the same way for each candidate, plus cost per 1,000 requests and p95 latency. Also weigh context length, tool-use reliability, structured-output support, data residency, licence (for open weights) and provider reliability.

Revisit regularly: prices drop and new models arrive constantly. Keep a model-agnostic client and an eval harness so switching is a measured, one-afternoon decision.