Statistics and the data stack

A random variable maps outcomes to numbers. Discrete ones have a probability mass function (PMF); continuous ones a probability density function (PDF), where only intervals have non-zero probability: $P(X = 1.5) = 0$, but $P(1 < X < 2)$ is the area under the curve.

The expectation $E[X]$ is the probability-weighted average. Variance is the expected squared distance from the mean; its square root is the standard deviation. Squaring penalises large deviations heavily, for exactly the same reason mean squared error does.

Distributions to know: Bernoulli (one yes/no trial: is this answer correct?), Binomial (count of successes in n trials: correct answers on an eval set), Normal (sums and averages of many things, know it cold), Uniform, and Poisson for counts per interval.

$P(A\mid B)$ is the probability of A given that B happened: restrict the world to B, then ask how much of it is A. By definition $P(A\mid B) = P(A \cap B)/P(B)$.

Write the joint probability both ways, $P(A\cap B) = P(A\mid B)P(B) = P(B\mid A)P(A)$, and divide: that is Bayes' theorem. It flips a conditional around: from 'how likely is this evidence if the hypothesis is true' to 'how likely is the hypothesis given the evidence'.

The classic trap is ignoring the base rate. A 99%-accurate detector for something that affects 1 in 1,000 people still produces mostly false positives. The simulation lets you see it as a population grid.

Take many samples of size n from almost any distribution and compute each sample's mean. The Central Limit Theorem says those means form an approximately normal distribution centred on the true mean, with spread $\sigma/\sqrt{n}$, the standard error.

That $\sqrt{n}$ is why precision is expensive: to halve your uncertainty you need four times the data.

It is also why most statistical tests work at all: we can reason about the sampling distribution of a mean even when the raw data is skewed. Watch the histogram of means turn into a bell curve in the simulation, even from a lopsided source.

A 95% confidence interval is a procedure that captures the true value 95% of the time across repeated samples. For a proportion $p$ on $n$ examples, it is roughly $p \pm 1.96\sqrt{p(1-p)/n}$.

A hypothesis test sets a null hypothesis $H_0$ (no difference) and asks: if $H_0$ were true, how likely is a result at least this extreme? That probability is the p-value. Small p means the data would be surprising under the null.

What a p-value is not: the probability the null is true, or the probability your result is a fluke, or a measure of effect size. A tiny, useless effect can have p < 0.001 with enough data; a large, important effect can miss p < 0.05 with too little.

The question you will face constantly: version B scored 3 points higher than A on your eval set. Is that real? With 50 examples, the standard error of an accuracy near 80% is about $\sqrt{0.8\cdot0.2/50} \approx 5.7$ points, so a 3-point gap is well inside the noise.

Better designs: use more examples; use a paired comparison (both versions on the same examples, then look at items where they disagree, as in McNemar's test); and bootstrap confidence intervals by resampling the eval set.

Decide the metric and sample size before looking at results, and keep a held-out set you don't tune against.

NumPy arrays are typed, contiguous blocks of memory. Vectorised operations (a * b, a @ b, np.exp(a)) run in optimised C, often 100× faster than Python loops.

Broadcasting lines up shapes from the right; a dimension of size 1 is stretched to match. So a (1000, 768) matrix minus a (768,) mean vector subtracts the mean from every row, with no loop and no copy.

This is how you compute all pairwise cosine similarities between queries and documents in one line: normalise the rows, then Q @ D.T.

A DataFrame is a table of typed columns. The workhorses: boolean filtering (df[df.age > 30]), groupby(...).agg(...) for split-apply-combine, merge for SQL-style joins, pivot_table and melt for reshaping, and .isna() for missing data.

EDA is not 'run df.describe()'. Look at distributions (histograms, box plots), relationships (scatter plots, correlation heatmaps), and especially subgroups: a pattern that holds overall may reverse within groups (Simpson's paradox).

A non-obvious finding is something like: one subgroup has a very different distribution, a correlation contradicts intuition, or the way data was collected created an artefact (all missing values come from one source). Those findings drive feature engineering in weeks 7–8.

Covariance measures whether two variables move together, in their own units. Pearson correlation rescales covariance to [−1, 1], capturing linear association. Spearman correlation uses ranks, catching any monotonic relationship and resisting outliers.

Correlation can be zero for a strong non-linear relationship (y = x² on symmetric data), and high for unrelated variables that share a cause (ice-cream sales and drowning both rise in summer). Always plot the scatter.

Causal claims need an experiment (randomised A/B test) or careful causal reasoning about confounders. In evals: 'longer answers score higher' may mean the judge prefers length, not that the answers are better.