Machine Learning Fundamentals · Unit 3 of 8
Unit 3: Training, testing and k-nearest neighbours
Why you hold back test data, training, validation and test sets, the k-nearest neighbours method by hand and in a lab, choosing k, and where k-NN struggles.
Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.
Lesson 1: Train, validate, test
Goal: Split data into training, validation and test sets, and split time data by time.
You cannot judge a model on the examples it learned from: it may simply have memorized them. So before training, hold some data back. The training set is what the model learns from. The test set is locked away and used once, at the end, to estimate how the model will do on new data.
If you also need to choose settings, such as k in k-NN or the depth of a tree, use a third piece, the validation set, to compare the options. Choosing settings by looking at the test set quietly turns it into training data, and your final score becomes too optimistic.
| Set | Typical share | Used for |
|---|---|---|
| Training | 60 to 80% | Fitting the model |
| Validation | 10 to 20% | Choosing settings and comparing models |
| Test | 10 to 20% | One final, honest estimate |
Worked example: when a random split is wrong
- A shop predicts next week's sales from two years of daily data.
- A random split puts some days from December 2025 in training and their neighbors in the test set. The model "predicts" days whose neighbors it has already seen.
- A time-based split trains on everything before a cut-off date and tests on everything after it, exactly as real forecasting works.
Split the way the model will be used. For data over time, train on the past and test on the future.
- Training set
- The examples a model learns from.
- Validation set
- Held-back examples used to choose settings and compare models.
- Test set
- Held-back examples used once, at the end, to estimate performance on new data.
Remember: Train on one part, tune on another, test once on a third. Time data: past for training, future for testing.
Lesson 2: k-nearest neighbours
Goal: Classify a point by hand with k-NN and Euclidean distance.
k-nearest neighbours (k-NN) is one of the simplest classifiers, and it makes "learning from examples" very concrete. To classify a new point, find the k training examples closest to it and let them vote. The majority label wins.
"Closest" usually means straight-line distance, called Euclidean distance. For two features: distance = √((x₁ − x₂)² + (y₁ − y₂)²). With more features you add more squared differences under the square root.
Worked example: classifying a tomato by hand
Features: redness (0 to 10) and firmness (0 to 10). Training examples: A ripe (8, 3), B ripe (7, 4), C unripe (2, 8), D unripe (3, 6), E ripe (6, 5). New tomato: (5, 5).
- Distance to A: √(3² + 2²) = √13 ≈ 3.61. To B: √(2² + 1²) = √5 ≈ 2.24.
- To C: √(3² + 3²) = √18 ≈ 4.24. To D: √(2² + 1²) = √5 ≈ 2.24. To E: √(1² + 0²) = 1.
- With k = 3 the nearest are E (ripe), B (ripe) and D (unripe): two votes to one.
- Prediction: ripe.
There is no real training step beyond storing the examples. All the work happens at prediction time.
Hands-on lab in the room: k-NN classifier. Classify new tomatoes, change k, and switch feature scaling on and off. Accuracy is measured on a held-back test set.
- k-nearest neighbours (k-NN)
- A method that predicts by majority vote (or average) of the k most similar training examples.
- Euclidean distance
- Straight-line distance: the square root of the sum of squared differences between features.
Remember: Find the k nearest training points and let them vote. Distance is the square root of summed squared differences.
Lesson 3: Choosing k
Goal: Explain how k changes k-NN and choose k with a validation set.
k controls how many neighbours vote, and it changes the model's behavior a lot.
- k = 1: the prediction copies the single nearest example, so every noisy or mislabeled point creates its own little island. Training accuracy is perfect, since each training point is its own nearest neighbour, and test accuracy often suffers.
- Large k: many neighbours vote, so predictions are smooth and stable. Too large, and the majority class swamps everything. With k equal to the whole training set, the model always predicts the most common class: the baseline.
- In between: usually best. Try several values on the validation set and pick the best one.
| k | Training accuracy | Validation accuracy |
|---|---|---|
| 1 | 100% | 78% |
| 3 | 93% | 86% |
| 7 | 89% | 88% |
| 15 | 85% | 84% |
| 61 (every example) | 60% | 60% |
For two classes, use an odd k to avoid ties. And because k-NN compares raw distances, scale your features first (Unit 2), or the feature with the biggest numbers decides everything. A setting like k, which you choose rather than the model learning it, is called a hyperparameter.
- Hyperparameter
- A setting chosen by the person building the model, such as k or tree depth, rather than learned from the data.
Remember: k = 1 memorizes, k = everything predicts the majority. Choose k on validation data.
Lesson 4: Strengths and limits of k-NN
Goal: Use k-NN for regression and explain its weaknesses, including the curse of dimensionality.
| Strengths | Limits |
|---|---|
| Simple to understand and explain: "these 5 similar customers did X" | Slow predictions on big data: every prediction compares with every stored example |
| No training time | Needs scaled features |
| Works for classification and regression | Struggles with many features (the curse of dimensionality) |
| Improves as soon as you add examples | Irrelevant features add noise to every distance |
k-NN regression predicts a number by averaging the labels of the k nearest neighbours. To estimate a flat's rent, average the rents of the 5 most similar flats.
Worked example: k-NN regression
- A flat with 2 bedrooms, 60 m², 1 km from the station.
- The 3 most similar flats in the data rent for 1,150, 1,200 and 1,330 a month.
- Prediction: (1,150 + 1,200 + 1,330) ÷ 3 ≈ 1,226.67, about 1,230 a month.
The same neighbours idea, but averaging instead of voting.
The curse of dimensionality is worth understanding because it affects many methods. With 2 features, nearby points are easy to find. With 200 features, points spread out so much that every point is far from every other, and "nearest" stops meaning "similar". More features is not automatically better; useful features are.
- k-NN regression
- Predicting a number as the average label of the k nearest neighbours.
- Curse of dimensionality
- The way distances become less meaningful as the number of features grows.
Remember: k-NN: simple and explainable, but slow at scale, needs scaling, and dislikes many or irrelevant features.
Practise it in Machine Learning Fundamentals
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: k-NN classifier.
Every lesson is free: no ads, no lives and no paywall.