Mathematics: linear algebra and calculus

Geometrically, a vector is an arrow from the origin. Numerically, it is a list of coordinates telling you how much of each basis vector to combine. In 2D the standard basis is $\hat{i} = (1,0)$ and $\hat{j} = (0,1)$, so $(3, 2)$ means three steps along $\hat{i}$ plus two along $\hat{j}$.

A linear combination scales and adds vectors: $a\vec{v} + b\vec{w}$. The span is every point you can reach that way. If one vector is a multiple of another, they are linearly dependent and add no new direction.

In AI, an embedding is a vector with hundreds of dimensions. You cannot picture 768 dimensions, but the same operations (add, scale, measure angle) work identically.

The single most useful reframe in linear algebra: a matrix is not a grid of numbers, it is a transformation of space that keeps grid lines parallel and evenly spaced and the origin fixed: rotation, scaling, shear, reflection or projection.

Read a 2×2 matrix column by column: the first column is where $\hat{i}$ lands, the second is where $\hat{j}$ lands. Every other vector follows, because it is a combination of $\hat{i}$ and $\hat{j}$. Drag the basis vectors in the simulation and watch the whole grid follow.

Matrix–vector multiplication $A\vec{x}$ is therefore 'apply transformation $A$ to $\vec{x}$'. A neural network layer $W\vec{x} + \vec{b}$ is exactly this, followed by a non-linearity.

$AB$ means 'apply $B$, then $A$'. That is why the order matters ($AB \neq BA$ in general) and why the row-times-column rule exists: it is what composition works out to. Shapes must chain: $(m\times n)(n\times p) = (m\times p)$.

The determinant is how much a transformation scales area (2D) or volume (3D). A negative determinant flips orientation; a zero determinant squashes space into a lower dimension, so the transformation can't be undone (no inverse).

Rank is the number of dimensions left after the transformation: the dimension of the column space. A 1000×1000 matrix of rank 8 can be written as the product of a 1000×8 and an 8×1000 matrix. That single fact is the whole idea behind LoRA in week 33.

Algebraically, the dot product multiplies matching coordinates and sums: $\vec{a}\cdot\vec{b} = \sum_i a_i b_i$. Geometrically, it equals $\|\vec{a}\|\|\vec{b}\|\cos\theta$: positive when the vectors point the same way, zero when perpendicular, negative when opposed.

Divide by both lengths and you get cosine similarity, which depends only on the angle. This is the standard measure of semantic similarity between embeddings, because direction encodes meaning and length often encodes things you don't care about (like text length).

If embeddings are already normalised to length 1, cosine similarity is the dot product, which is why many vector databases normalise once at insert time and then use the cheaper dot product.

Most vectors get knocked off their span when you apply a matrix. Eigenvectors are the special ones that stay on their own line: $A\vec{v} = \lambda\vec{v}$. The eigenvalue $\lambda$ says how much they stretch (or flip, if negative).

They reveal the 'natural axes' of a transformation. PCA (week 10) finds the eigenvectors of the data's covariance matrix: the directions of greatest variance. Repeated application of a matrix is dominated by its largest eigenvalue, which is why gradients in deep RNNs explode or vanish (week 18).

SVD generalises this to any matrix: every matrix is a rotation, then a scaling along axes, then another rotation ($U\Sigma V^T$). Keeping only the largest singular values gives the best low-rank approximation.

The derivative $f'(x)$ is the slope of $f$ at $x$: how much the output changes per tiny change in input. Rules worth knowing cold: power rule $\frac{d}{dx}x^n = nx^{n-1}$, exponentials $\frac{d}{dx}e^x = e^x$, and the sigmoid's tidy derivative $\sigma'(x) = \sigma(x)(1-\sigma(x))$.

The chain rule: if $y = f(g(x))$, then $\frac{dy}{dx} = f'(g(x))\cdot g'(x)$. Wiggle $x$; it moves $g$ by $g'$; that moves $f$ by $f'$ times as much. Rates multiply along the chain.

A neural network is a long chain of functions. Backpropagation is nothing more than applying the chain rule from the loss backwards through every step, reusing intermediate results.

For a function of many inputs, a partial derivative measures the slope along one input while holding the others fixed. Stack all of them and you get the gradient $\nabla L$, a vector pointing in the direction of steepest increase.

Gradient descent minimises a loss by repeatedly stepping against the gradient: $\theta \leftarrow \theta - \alpha\nabla L(\theta)$. The learning rate $\alpha$ sets the step size: too small and training crawls; too large and it overshoots and diverges. Try both in the simulation.

For linear regression with mean squared error, the gradients have a closed form you can derive by hand, which is this week's capstone.