Machine Learning Fundamentals · Unit 6 of 8
Unit 6: Overfitting and regularization
Underfitting and overfitting, finding the sweet spot with validation and cross-validation, L1 and L2 regularization, and other defenses, including never peeking at the test set.
Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.
Lesson 1: Underfitting and overfitting
Goal: Diagnose underfitting and overfitting from training and test error.
Underfitting: the model is too simple to capture the pattern, so it does badly on training data and test data alike. A straight line through data that clearly curves is the classic case.
Overfitting: the model is so flexible that it memorizes the training data, noise and all. Training error is tiny; test error is large. The gap between the two is the warning sign.
| Training error | Test error | What to do | |
|---|---|---|---|
| Underfit | High | High | A more flexible model or better features |
| Good fit | Low | Low, close to training | Keep it |
| Overfit | Very low | Much higher | A simpler model, more data, regularization |
Worked example: three curves through the same points
Twelve noisy measurements that follow a gentle curve.
- A straight line (degree 1) misses the bend: training MSE 4.1, test MSE 4.6. Underfit.
- A smooth curve (degree 3) follows the bend: training MSE 1.2, test MSE 1.5. A good fit.
- A wiggly curve (degree 11) passes through every training point: training MSE 0.0, test MSE 19.8. It learned the noise.
Perfect performance on the training data is a warning, not a victory.
- Underfitting
- A model too simple to capture the real pattern: high error everywhere.
- Overfitting
- A model that memorizes training data, including its noise, and does much worse on new data.
- Noise
- Random variation in data that does not reflect the real pattern.
Remember: High error everywhere: underfit. Low training error, high test error: overfit. Watch the gap.
Lesson 2: Finding the sweet spot
Goal: Choose model complexity with validation data, cross-validation and learning curves.
Model complexity, such as the degree of a curve, the depth of a tree or the number of features, is a dial. Turn it up and training error keeps falling, while test error first falls and then rises again. The best model sits at the bottom of the test-error curve, and you find it with a validation set, never the test set.
Hands-on lab in the room: Overfitting and regularization demo. Change the degree of a curve and watch training and test error. Then add regularization to tame a very flexible curve.
Cross-validation makes better use of limited data. In k-fold cross-validation you split the training data into k parts (folds), train k times, each time holding out a different fold for validation, and average the k scores. With 5 folds, every example is used for validation exactly once.
Worked example: 5-fold cross-validation
- 500 training examples are split into 5 folds of 100.
- Run 1 trains on folds 2 to 5 and validates on fold 1; run 2 validates on fold 2; and so on.
- The five validation scores are 84%, 86%, 83%, 85% and 87%: an average of 85%.
- Repeat for each setting you are comparing, choose the best average, then train on all 500 and test once on the separate test set.
A steadier estimate than one validation split, at the cost of training k times.
More data is the most reliable cure for overfitting: with more examples, there is less room to memorize noise. A learning curve, error plotted against training-set size, shows whether more data would help. If test error is still falling as the data grows, collecting more is worth it.
- Model complexity
- How flexible a model is, for example the degree of a curve or the depth of a tree.
- Cross-validation
- Training and validating several times on different splits of the training data, then averaging the scores.
- Learning curve
- A plot of error against the amount of training data.
Remember: Turn the complexity dial while watching validation error. Cross-validate when data is scarce.
Lesson 3: Regularization
Goal: Explain L1 and L2 regularization and calculate a regularized loss.
Regularization adds a penalty for complexity to the loss, so the model has to justify large weights with a real improvement in fit. Wild, wiggly curves need huge weights, and the penalty makes them expensive.
- L2 (ridge): loss = MSE + λ × (sum of squared weights). It shrinks all weights smoothly toward zero.
- L1 (lasso): loss = MSE + λ × (sum of absolute weights). It tends to push some weights exactly to zero, which removes features and makes the model simpler to read.
λ (lambda) sets the strength. λ = 0 means no regularization. Too large, and the model is forced toward flat, timid predictions: underfitting. Like k and tree depth, λ is a hyperparameter you choose with validation data.
Worked example: the penalty in numbers
Two candidate models for the same data, with λ = 0.5 and an L2 penalty.
- Model A: MSE 2.0, weights 1 and −2. Penalty = 0.5 × (1 + 4) = 2.5, total 4.5.
- Model B: MSE 1.6, weights 4 and −5. Penalty = 0.5 × (16 + 25) = 20.5, total 22.1.
- B fits the training data slightly better but pays a huge price for its large weights.
- Training with regularization prefers A, which usually generalizes better.
Regularization trades a little training fit for a lot of stability.
- Regularization
- Adding a penalty for large weights to the loss, to discourage overfitting.
- L2 (ridge) regularization
- A penalty on the sum of squared weights.
- L1 (lasso) regularization
- A penalty on the sum of absolute weights, which can set some weights to zero.
- Lambda (λ)
- The strength of the regularization penalty.
Remember: Loss = fit + λ × size of weights. L2 shrinks, L1 can zero out. Tune λ on validation data.
Lesson 4: Other defenses, and peeking
Goal: Use early stopping and other defenses, and protect the test set.
- Simpler models: fewer features, a lower degree, a shallower tree.
- Early stopping: when training step by step, watch validation loss and stop when it starts rising, even though training loss is still falling.
- More and better data: more examples, corrected labels, and augmented examples (for images, flipped or cropped copies).
- Dropout: in neural networks, randomly switching off some units during training so the network cannot lean on any single one (Unit 8 explains networks).
- Ensembles: averaging many models, such as a random forest (Unit 7).
| Epoch | Training loss | Validation loss |
|---|---|---|
| 5 | 0.90 | 0.95 |
| 10 | 0.55 | 0.62 |
| 15 | 0.40 | 0.58 |
| 20 | 0.31 | 0.61 |
| 30 | 0.18 | 0.74 |
Watch out: Overfitting to the test set: every time you look at the test score and change something, a little information leaks from the test set into your choices. Do it 50 times and your "test" score is optimistic. Make decisions with validation data, and touch the test set once, at the end. Machine learning competitions hide their final test sets for exactly this reason.
A useful habit is to write down, before you start, what you will report and how: which metric, which test set, which baseline. It keeps you honest when a result disappoints and the temptation to tweak "just once more" arrives.
- Early stopping
- Stopping training when validation loss stops improving.
- Epoch
- One full pass through the training data during training.
- Dropout
- Randomly disabling some neural network units during training to reduce overfitting.
Remember: Stop when validation loss turns up. Decide with validation data. Use the test set once.
Practise it in Machine Learning Fundamentals
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Overfitting and regularization demo.
Every lesson is free: no ads, no lives and no paywall.