Eigenvectors

Most vectors get knocked off their span when you apply a matrix. Eigenvectors are the special ones that stay on their own line: $A\vec{v} = \lambda\vec{v}$. The eigenvalue $\lambda$ says how much they stretch (or flip, if negative).

They reveal the 'natural axes' of a transformation. PCA (week 10) finds the eigenvectors of the data's covariance matrix: the directions of greatest variance. Repeated application of a matrix is dominated by its largest eigenvalue, which is why gradients in deep RNNs explode or vanish (week 18).

SVD generalises this to any matrix: every matrix is a rotation, then a scaling along axes, then another rotation ($U\Sigma V^T$). Keeping only the largest singular values gives the best low-rank approximation.