Week 9: ML algorithms (1/2) · AI Engineer Path

Concepts

6 lessons · tick each one once you could explain it

In plain words

Linear regression predicts a number as a weighted sum of the inputs, plus a constant: price = a × size + b × rooms + c. It chooses the weights that make the squared prediction errors as small as possible. Each weight says how much the prediction changes per unit of that feature, if everything else stays the same.

Drawing the line through a scatter of points that keeps them as close to it as possible, counting each miss as its distance squared.

In detail

The model predicts $\hat y = w^\top x + b$ . Ordinary least squares picks the weights that minimise mean squared error. There is a closed-form solution (the normal equation), but gradient descent scales better and is what you already built in week 4.

Coefficients are interpretable: holding other features fixed, a one-unit change in $x_j$ changes the prediction by $w_j$ . That interpretation breaks under multicollinearity (correlated features share credit unpredictably).

Assumptions worth checking via residual plots: linearity, constant variance of errors, independent errors.

\hat{w} = (X^\top X)^{-1} X^\top y

$\hat{w}$: the best-fitting weights
$X$: the data matrix: one row per example, one column per feature
$y$: the vector of true target values
$X^\top$: $X$ transposed (rows and columns swapped)
$(\cdot)^{-1}$: the matrix inverse

Worked example

Fitting a line by hand to four points

Points (1, 2), (2, 4), (3, 5), (4, 9). Means: x̄ = 2.5, ȳ = 5.
Slope: $w = \sum (x - \bar x)(y - \bar y) / \sum (x - \bar x)^2 = 11 / 5 = 2.2$ .
Intercept: $b = \bar y - w\bar x = 5 - 2.2 \times 2.5 = -0.5$ .
Predictions: 1.7, 3.9, 6.1, 8.3. Residuals: 0.3, 0.1, −1.1, 0.7, which sum to zero, as least squares guarantees.
Reading it: each extra unit of x adds 2.2 to the predicted y.

Least squares picks the line that minimises squared residuals; each weight is the effect of one feature, holding the others fixed.

Common mistakes

Reading coefficients as effects when features are strongly correlated: credit splits between them unpredictably.
Ignoring residual plots: a curve or a funnel shape in the residuals means the model is missing structure.

Check yourself

A house-price model has weight 3,000 on size_m2. What does that mean?Show answer

Holding the other features fixed, each extra square metre adds 3,000 to the predicted price.

Why do libraries avoid computing

(X^\top X)^{-1}

directly?Show answer

It's slow for many features and numerically unstable when features are collinear. QR or SVD decompositions, or gradient descent, are more reliable.

Going deeper

The normal equation costs O(d³) and is numerically fragile when features are collinear; libraries use QR or SVD instead. Gradient descent and SGD scale to huge datasets.

Linear models are interpretable only when features are independent-ish and on known scales. Standardise and check variance inflation factors before reading coefficients as 'effects'.

Best resources for this lesson

InteractiveLinear regression (MLU-Explain)
BookAn Introduction to Statistical Learning (free book) · chapter 3; the best gentle-but-rigorous ML text

Where this comes back

Week 4You wrote its gradient descent by hand in the maths capstone.

In plain words

Regularisation adds a penalty for large weights to the training loss. The model then prefers simpler explanations, which makes it less likely to fit noise. Ridge (L2) shrinks all weights smoothly; Lasso (L1) can push some weights to exactly zero, dropping those features entirely.

A budget on how much the model may 'spend' on weights: Ridge taxes big spending heavily, Lasso charges a flat fee per unit, so small, unimportant purchases get cut altogether.

In detail

Ridge (L2) adds $\lambda\sum w_j^2$ to the loss: it shrinks all weights smoothly and handles correlated features gracefully. Lasso (L1) adds $\lambda\sum |w_j|$ : it drives some weights exactly to zero, performing feature selection. ElasticNet mixes both.

The strength $\lambda$ (scikit-learn's alpha, or C = 1/λ in classifiers) is a hyperparameter chosen by cross-validation. Scale features first, or the penalty treats them unequally.

In neural networks, L2 regularisation appears as weight decay (AdamW, week 16).

L_{\text{ridge}} = \text{MSE} + \lambda\sum_j w_j^2, \qquad L_{\text{lasso}} = \text{MSE} + \lambda\sum_j |w_j|

$L_{\text{ridge}},\ L_{\text{lasso}}$: the training loss with each penalty added
$\text{MSE}$: the usual mean squared error
$\lambda$: the penalty strength: 0 means no regularisation
$w_j$: the weight on feature $j$

Worked example

Shrinking the slope from the four-point example

From the linear regression example: $\sum(x-\bar x)(y-\bar y) = 11$ and $\sum(x-\bar x)^2 = 5$ , giving slope 2.2 with no penalty.
Ridge, using summed squared error plus $\lambda w^2$ : the best slope becomes $11 / (5 + \lambda)$ . At λ = 5 it's 1.1; at λ = 45 it's 0.22. Smaller, but never exactly zero.
Lasso, using summed squared error plus $\lambda |w|$ : the slope becomes $(11 - \lambda/2) / 5$ . At λ = 10 it's 1.2; at λ = 22 or more it's exactly 0.
That's why Lasso performs feature selection: weak features hit zero.
Choose λ by cross-validation, for example with RidgeCV or LassoCV, and scale features first.

Ridge shrinks weights towards zero; Lasso can set them exactly to zero. The strength λ is tuned by cross-validation.

Common mistakes

Regularising unscaled features: the penalty then hits features unequally, depending on their units.
Confusing the knobs: in scikit-learn classifiers, C is the inverse of strength, so a smaller C means more regularisation.

Check yourself

You have 500 features and suspect only 20 matter. Ridge or Lasso?Show answer

Lasso (or ElasticNet): it drives the weights of unhelpful features to exactly zero, effectively selecting the useful ones.

What happens as λ grows very large?Show answer

All weights are pushed towards zero and the model predicts close to a constant: high bias, low variance.

Going deeper

Geometrically, L1's diamond-shaped constraint has corners on the axes, so the optimum often lands where some weights are exactly zero; L2's circle doesn't. From a Bayesian view, L2 is a Gaussian prior on weights and L1 a Laplace prior.

Use RidgeCV, LassoCV or ElasticNetCV to tune strength efficiently along a regularisation path.

Best resources for this lesson

DocsLinear models (scikit-learn)
VideoRidge regression (StatQuest)

In plain words

Logistic regression answers yes/no questions with a probability. It computes a weighted score from the inputs, just like linear regression, then squashes it through the S-shaped sigmoid curve into a number between 0 and 1. Training punishes confident wrong answers very heavily.

The sigmoid is a dimmer switch: very negative scores are 'off' (near 0), very positive are 'on' (near 1), with a smooth fade in between.

In detail

Logistic regression computes a linear score $z = w^\top x + b$ and turns it into a probability with the sigmoid $\sigma(z) = 1/(1+e^{-z})$ . The decision boundary ( $z = 0$ ) is a straight line (a hyperplane in higher dimensions).

It is trained by minimising log loss (binary cross-entropy), which heavily punishes confident wrong answers. With more classes, the sigmoid becomes a softmax and the loss becomes categorical cross-entropy.

That combination (linear layer + softmax + cross-entropy) is exactly the output layer of a neural classifier and of an LLM predicting the next token.

P(y=1\mid x) = \sigma(w^\top x + b),\qquad L = -\big[y\log p + (1-y)\log(1-p)\big]

$P(y=1\mid x)$: the predicted probability that the answer is 'yes'
$\sigma$: the sigmoid, $1/(1+e^{-z})$ , which maps any number into (0, 1)
$w^\top x + b$: the linear score $z$ : weighted sum of features plus bias
$p$: the predicted probability
$y$: the true label, 1 or 0
$L$: log loss (binary cross-entropy)

Worked example

Scoring one email for spam

Weights: w = 2, b = −1. The email's feature value is x = 1.5.
Score: $z = 2 \times 1.5 - 1 = 2$ . Probability: $\sigma(2) = 1/(1 + e^{-2}) \approx 0.88$ .
If it really is spam (y = 1), log loss is $-\ln 0.88 \approx 0.13$ : small.
If it isn't spam (y = 0), log loss is $-\ln 0.12 \approx 2.1$ . A confident 0.99 that turns out wrong costs $-\ln 0.01 \approx 4.6$ .
The weight 2 means each unit of x multiplies the odds of spam by $e^2 \approx 7.4$ .

Linear score, then sigmoid for a probability, trained with log loss, which punishes confident mistakes hardest.

Common mistakes

Treating raw scores as calibrated probabilities after heavy regularisation or rebalancing. Check calibration.
Expecting curved decision boundaries: the boundary is a straight line (hyperplane) unless you add non-linear features.

Check yourself

What is

\sigma(0)

, and what does a score of 0 mean?Show answer

$\sigma(0) = 0.5$ : the model is exactly undecided. Points with score 0 lie on the decision boundary.

How does logistic regression relate to an LLM's output layer?Show answer

Same recipe, more classes: a linear layer produces scores for every token in the vocabulary, softmax turns them into probabilities, and cross-entropy trains them.

Going deeper

Coefficients are log-odds: a coefficient of 0.7 multiplies the odds by e^0.7 ≈ 2 per unit increase. That makes logistic regression a favourite in regulated domains where decisions must be explained.

With perfectly separable data the unregularised optimum is at infinity (weights grow without bound), which is another reason regularisation is the default.

Best resources for this lesson

InteractiveLogistic regression (MLU-Explain)
BookSpeech and Language Processing, ch. 5: logistic regression · Jurafsky & Martin, free draft

Where this comes back

Week 15Softmax + cross-entropy is the standard neural classification head.
Week 13TF-IDF + logistic regression is your text baseline.

In plain words

A decision tree makes predictions by asking a series of yes/no questions about the features, like a flowchart. At each step it picks the question that best separates the classes, measured by how 'pure' the resulting groups are. Trees are easy to read but memorise their training data if allowed to grow too deep.

The game of Twenty Questions: each good question splits the remaining possibilities as cleanly as possible.

In detail

A tree asks yes/no questions about features (income > 50k?), splitting data into regions with a constant prediction. Each split is chosen greedily to maximise purity gain, measured by Gini impurity or entropy for classification, and variance reduction for regression.

Trees handle mixed feature types, need no scaling, capture interactions and are readable. Alone, they overfit badly: a deep tree memorises the training data. Control it with max_depth, min_samples_leaf or cost-complexity pruning.

Their real power is as building blocks for random forests and gradient boosting (week 10). Watch a tree carve up 2D space in the simulation.

\text{Gini} = 1 - \sum_k p_k^2, \qquad H = -\sum_k p_k \log_2 p_k

$p_k$: the share of examples in the node that belong to class $k$
$\text{Gini}$: the chance two random picks from the node disagree: 0 = pure
$H$: entropy, in bits: 0 = pure, 1 = a 50/50 split of two classes

Worked example

Choosing a split for 10 emails

6 spam and 4 not spam. Gini impurity: $1 - (0.6^2 + 0.4^2) = 1 - 0.52 = 0.48$ .
Candidate question: 'contains the word free?' Yes branch: 5 emails, all spam (Gini 0). No branch: 1 spam, 4 not spam (Gini $1 - (0.2^2 + 0.8^2) = 0.32$ ).
Weighted impurity after the split: 0.5 × 0 + 0.5 × 0.32 = 0.16. The improvement is 0.48 − 0.16 = 0.32.
The tree compares this gain with every other possible question and picks the best, then repeats inside each branch.
For comparison, the root's entropy is $-(0.6\log_2 0.6 + 0.4\log_2 0.4) \approx 0.97$ bits.

Trees greedily pick the question that most reduces impurity, then recurse; depth limits stop them memorising.

Common mistakes

Growing trees without limits: they memorise the training set. Set max_depth or min_samples_leaf, or prune.
Using a tree to extrapolate: beyond the training range it predicts a constant.

Check yourself

What is the Gini impurity of a node with 50% of each of two classes?Show answer

$1 - (0.5^2 + 0.5^2) = 0.5$ , the maximum for two classes.

Why don't trees need feature scaling?Show answer

Each split compares one feature against a threshold, and rescaling the feature just moves the threshold. The possible splits are the same.

Going deeper

Trees are unstable: small data changes can produce a completely different tree. That instability is exactly why averaging many trees (random forests) works so well.

Trees extrapolate poorly: outside the training range they predict a constant. Keep that in mind for forecasting or any feature that drifts upward over time.

Best resources for this lesson

InteractiveDecision trees (MLU-Explain)
DocsDecision trees (scikit-learn)

In plain words

k-nearest neighbours predicts by finding the k most similar training examples and letting them vote. Naive Bayes multiplies together how strongly each feature points to each class, assuming (naively) that features are independent. Both are simple, fast to set up and surprisingly strong baselines.

kNN: to guess a stranger's taste in films, ask the three people most like them. Naive Bayes: tally the evidence clue by clue, as if each clue were independent.

In detail

kNN stores the training set and predicts by majority vote (or average) of the k closest points. No training, but slow prediction and sensitive to scaling and to the curse of dimensionality. It is the conceptual ancestor of vector search: retrieval in RAG is kNN over embeddings.

Naive Bayes applies Bayes' theorem and assumes features are conditionally independent given the class. The assumption is almost always false, yet it works well for text classification because it is fast, needs little data and is robust.

Both make strong, cheap baselines.

P(c\mid x_1,\dots,x_n) \propto P(c)\prod_{i} P(x_i \mid c)

$c$: a class, such as 'spam'
$x_1,\dots,x_n$: the observed features
$P(c)$: how common the class is overall (the prior)
$\prod_i P(x_i \mid c)$: multiply how likely each feature is within that class
$\propto$: 'proportional to': normalise the scores so they sum to 1

Worked example

Classifying an email both ways

kNN with k = 3: the three most similar training emails are spam, spam and not spam, so predict spam.
Naive Bayes priors: P(spam) = 0.4 and P(ham) = 0.6. The email contains 'free' and 'meeting'.
From the training data: P(free | spam) = 0.5, P(free | ham) = 0.05, P(meeting | spam) = 0.05, P(meeting | ham) = 0.3.
Spam score: 0.4 × 0.5 × 0.05 = 0.010. Ham score: 0.6 × 0.05 × 0.3 = 0.009.
Normalise: P(spam | email) = 0.010 / 0.019 ≈ 53%. A close call: 'free' points to spam, 'meeting' to ham.

kNN lets similar examples vote; Naive Bayes multiplies per-feature evidence. Both make cheap, solid baselines.

Common mistakes

Running kNN on unscaled features, so one large-unit feature decides every 'nearest' neighbour.
Trusting Naive Bayes probabilities: they're often overconfident, even when its ranking of classes is good.

Check yourself

How is retrieval in a RAG system related to kNN?Show answer

It is kNN: find the k stored chunks whose embeddings are nearest the query's embedding.

Why does Naive Bayes work for spam despite the independence assumption being false?Show answer

Classification only needs the right class to score highest. Even with distorted probabilities, the ranking is often correct, and the model is fast and needs little data.

Going deeper

kNN's prediction quality depends entirely on the distance metric and feature scaling. With learned embeddings, kNN becomes powerful again: it's how few-shot example selection and RAG retrieval work.

Naive Bayes is often poorly calibrated (overconfident) because correlated features are double-counted, but its ranking of classes is frequently still good.

Best resources for this lesson

Where this comes back

Week 13Naive Bayes is a standard text-classification baseline.
Week 30Vector search is approximate kNN at scale.

In plain words

A support vector machine draws the boundary between two classes that leaves the widest possible gap on either side. Only the points closest to the boundary, the support vectors, decide where it goes. Kernels let that boundary curve by implicitly working in a richer feature space.

Laying the widest possible road between two villages: only the houses at the edge of each village decide where the road can run.

In detail

An SVM finds the hyperplane that separates classes with the maximum margin. Only the points on the edge of the margin (the support vectors) define it. The C parameter trades margin width against misclassified training points.

The kernel trick computes similarity in a high-dimensional feature space without ever building it, so a linear separator there becomes a curved boundary in the original space. RBF is the common default.

SVMs shine on medium-sized, high-dimensional data (like text before deep learning) but scale poorly to very large datasets.

\min_{w,b}\ \tfrac12\|w\|^2 + C\sum_i \max\big(0,\ 1 - y_i(w^\top x_i + b)\big)

$w,\ b$: the boundary's direction and offset
$\tfrac12\|w\|^2$: a small $\|w\|$ means a wide margin, so minimising it widens the gap
$C$: how much to punish points inside the margin or misclassified
$y_i$: the label of point $i$ : +1 or −1
$\max(0,\ 1 - y_i(w^\top x_i + b))$: the hinge loss: zero for points safely outside the margin

Worked example

Margins with w = (1, 1) and b = −3

The boundary is where $x_1 + x_2 - 3 = 0$ . The margin edges are where the score is +1 and −1.
Point (2, 2), class +1: score 2 + 2 − 3 = 1, exactly on its margin edge, so it's a support vector.
Point (1, 1), class −1: score −1, on the other edge, also a support vector.
Margin width: $2/\|w\| = 2/\sqrt{2} \approx 1.41$ .
Point (2.5, 1), class +1: score 0.5, inside the margin. Hinge loss $\max(0, 1 - 0.5) = 0.5$ , scaled by C in the objective.

Maximise the margin, pay a hinge penalty (weighted by C) for points inside it, and let kernels bend the boundary.

Common mistakes

Using kernel SVMs on hundreds of thousands of rows: training scales roughly quadratically or worse.
Setting RBF gamma too high, which makes wiggly boundaries that memorise the training data.

Check yourself

What does a large C do?Show answer

Punishes margin violations heavily, so the SVM tries hard to classify every training point: a narrower margin and more risk of overfitting.

What does the kernel trick buy you?Show answer

Similarities computed as if the data were mapped into a high-dimensional space, without building that space, so a straight separator there becomes a curved one in the original space.

Going deeper

The hinge loss only penalises points inside the margin or misclassified, which is why only support vectors matter. The RBF kernel's gamma controls how far each training point's influence reaches: high gamma means wiggly boundaries and overfitting.

Linear SVMs (LinearSVC) scale well to high-dimensional sparse text; kernel SVMs scale roughly quadratically or worse with samples.

Best resources for this lesson

Practice

Hands-on work that makes the lessons stick. Warm-ups take minutes; stretch goals are optional.

Warm-up
Regularisation paths
Plot Lasso and Ridge coefficients as the regularisation strength varies. Watch Lasso zero out features one by one.
Core
Algorithm bake-off
On one classification dataset, compare logistic regression, kNN, Naive Bayes, SVM and a decision tree with the same CV. Explain the ranking.
Stretch
Logistic regression from scratch
Implement logistic regression with gradient descent in NumPy and match scikit-learn's coefficients.

This week, day by day

Dates follow your pace from Settings. Open the notebook icon to log hours and notes.

Day 57Monday16 Nov2 h planned
Linear regression from scratch, then sklearn
Day 58Tuesday17 Nov2 h planned
Regularisation: Ridge, Lasso, ElasticNet
Day 59Wednesday18 Nov2 h planned
Logistic regression and decision boundaries
Day 60Thursday19 Nov2 h planned
Decision trees: splitting criteria, pruning, interpretation
Day 61Friday20 Nov2 h planned
kNN and Naive Bayes
Day 62Saturday21 Nov3 h planned
SVM: margins, the kernel trick
Day 63Sunday22 NovReview
Review the week, finish anything unfinished, rest

Watch

100 Days of Machine LearningPrimary

CampusX · playlist

The roadmap author's own course; node names map to its days.

Machine Learning

StatQuest · playlist

Machine Learning Specialization

Andrew Ng · DeepLearning.AI · playlist

Stanford CS229: Machine Learning (2022)

Stanford Online · playlist

Theory depth. Optional.

Linear Regression, Clearly Explained

StatQuest

Ridge (L2) Regression

StatQuest

Lasso (L1) Regression

StatQuest

Logistic Regression

StatQuest

Decision and Classification Trees

StatQuest

K-nearest neighbors

StatQuest

Naive Bayes, Clearly Explained

StatQuest

Support Vector Machines: Main Ideas

StatQuest

Read and use

Interview prep

Questions this week's material gets asked as. Answer out loud first, then open the outline.

How do L1 and L2 regularisation differ?

L1 drives some weights to exactly zero (selection)
L2 shrinks smoothly; handles correlated features
Both trade bias for variance; tune by CV

Why is logistic regression still used in industry?

Fast, stable, calibrated-ish probabilities
Interpretable coefficients (log-odds)
Strong baseline; easy to deploy and monitor

Check yourself

Five questions. The done-when test above is the real bar; this is a quick self-check.

1.Which regularisation can set weights exactly to zero?
2.Logistic regression's decision boundary is…
3.A single unpruned decision tree typically…
4.Why does kNN degrade in very high dimensions?
5.Small table (2,000 rows) with mixed types; you need a strong, explainable baseline. Good first pick?

Why this week matters

Done when

Concepts

Linear regression

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

Where this comes back

Regularisation: Ridge, Lasso, ElasticNet

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

Logistic regression and decision boundaries

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

Where this comes back

Decision trees

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

k-nearest neighbours and Naive Bayes

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

Where this comes back

Support vector machines and the kernel trick

In plain words

In detail

Worked example

Common mistakes

Check yourself

Going deeper

Best resources for this lesson

Practice

Regularisation paths

Algorithm bake-off

Logistic regression from scratch

This week, day by day

Watch

Read and use

Interview prep

Check yourself