Machine Learning Fundamentals · Unit 5 of 8
Unit 5: Classification and its metrics
From scores to decisions with logistic regression and thresholds, the confusion matrix, why accuracy misleads, precision and recall, and choosing a threshold from real costs.
Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.
Lesson 1: From scores to classes
Goal: Explain classifier scores, thresholds and logistic regression.
Many classifiers do not output a label directly. They output a score, often a probability between 0 and 1: "this email is 0.92 likely to be spam". A threshold turns the score into a decision: with a threshold of 0.5, anything scoring 0.5 or more is treated as spam.
Logistic regression is the classic example. It computes a weighted sum of the features, just like linear regression, then passes it through the sigmoid function, which squashes any number into the range 0 to 1. Large positive sums become scores near 1, large negative sums scores near 0, and a sum of exactly 0 becomes 0.5.
| Weighted sum | Sigmoid score |
|---|---|
| −4 | 0.02 |
| −1 | 0.27 |
| 0 | 0.50 |
| 1 | 0.73 |
| 4 | 0.98 |
Logistic regression is trained like linear regression, with gradient descent, but with a loss suited to probabilities (called log loss or cross-entropy) that punishes confident wrong answers heavily. The threshold, though, is not learned. It is a business decision, and the last lesson in this unit shows how moving it changes which mistakes you make.
- Score (probability)
- A classifier's number between 0 and 1 saying how strongly an example looks like the positive class.
- Threshold
- The score at or above which an example is labeled positive.
- Logistic regression
- A classifier that passes a weighted sum of features through the sigmoid to get a score.
- Sigmoid
- The S-shaped function that maps any number to a value between 0 and 1.
Remember: Weighted sum → sigmoid → score → threshold → decision. The threshold is your choice.
Lesson 2: The confusion matrix
Goal: Fill in a confusion matrix and calculate accuracy, and explain the accuracy paradox.
To evaluate a classifier, count its four kinds of outcome on the test set. Take a fraud model where "positive" means "flagged as fraud".
| Actually fraud | Actually fine | |
|---|---|---|
| Model says fraud | True positive (TP): caught | False positive (FP): false alarm |
| Model says fine | False negative (FN): missed | True negative (TN): correctly ignored |
Worked example: counting outcomes
1,000 test transactions, 20 of them really fraud. The model flags 30.
- Of the 30 flagged, 15 are fraud (TP = 15) and 15 are fine (FP = 15).
- Of the 20 frauds, 15 were caught, so 5 were missed (FN = 5).
- Everything else is a correct "fine": TN = 1,000 − 15 − 15 − 5 = 965.
- Accuracy = (TP + TN) ÷ total = (15 + 965) ÷ 1,000 = 98%.
98% sounds excellent, yet a quarter of the frauds got through.
This is the accuracy paradox: when one class is rare, accuracy is dominated by the easy majority. A model that flags nothing at all scores 980 ÷ 1,000 = 98% here too, and catches zero fraud. That is why professionals report precision and recall, the subject of the next lesson.
- Confusion matrix
- A table counting true positives, false positives, false negatives and true negatives.
- True positive (TP)
- A positive case correctly flagged.
- False positive (FP)
- A negative case wrongly flagged: a false alarm.
- False negative (FN)
- A positive case the model missed.
- True negative (TN)
- A negative case correctly left alone.
- Accuracy
- (TP + TN) ÷ all cases.
Remember: Count TP, FP, FN and TN before you trust any accuracy figure.
Lesson 3: Precision and recall
Goal: Calculate precision, recall and F1, and decide which matters in a situation.
Precision answers: when the model says positive, how often is it right? Precision = TP ÷ (TP + FP).
Recall answers: of all the real positives, how many did the model find? Recall = TP ÷ (TP + FN).
Worked example: the fraud model again
- TP = 15, FP = 15 and FN = 5.
- Precision = 15 ÷ (15 + 15) = 50%: half the flags are false alarms.
- Recall = 15 ÷ (15 + 5) = 75%: a quarter of the frauds slip through.
- The F1 score combines the two: 2 × P × R ÷ (P + R) = 2 × 0.5 × 0.75 ÷ 1.25 = 0.6.
Precision and recall tell you what kind of mistakes the model makes. Accuracy did not.
| Situation | Which matters more | Why |
|---|---|---|
| Spam filter | Precision | A real email lost in the spam folder is costly; a little spam in the inbox is not |
| First-stage screening for a serious disease, followed by a proper test | Recall | Missing a real case is far worse than an extra follow-up test |
| Fraud alerts that freeze cards | Both, balanced | Missed fraud costs money; false freezes anger customers |
| Recommending a film | Precision | Bad suggestions annoy; missing some good ones costs little |
- Precision
- TP ÷ (TP + FP): how often a positive prediction is right.
- Recall
- TP ÷ (TP + FN): the share of real positives the model finds.
- F1 score
- A single number combining precision and recall: 2PR ÷ (P + R).
Remember: Precision: are my alarms right? Recall: did I catch them all? Pick by the cost of each mistake.
Lesson 4: Thresholds and trade-offs
Goal: Move a threshold to trade precision against recall, and choose it from real costs.
Moving the threshold trades precision against recall. Lower it and the model flags more: recall rises (it finds more real positives) and precision usually falls (more false alarms). Raise it and the opposite happens. There is no free lunch: you are choosing which mistakes to make more of.
Worked example: choosing a threshold from costs
A missed fraud costs about 500; a false alarm costs about 20 in staff time and customer annoyance.
- At threshold 0.7: FN = 10 and FP = 20, so cost = 10 × 500 + 20 × 20 = 5,400.
- At threshold 0.4: FN = 4 and FP = 90, so cost = 4 × 500 + 90 × 20 = 3,800.
- At threshold 0.2: FN = 2 and FP = 300, so cost = 2 × 500 + 300 × 20 = 7,000.
- The middle threshold costs least.
Put real costs on each kind of mistake and the choice of threshold becomes a calculation, not a guess.
Hands-on lab in the room: Confusion matrix and threshold lab. Slide the threshold on a scored test set and watch the confusion matrix, precision and recall move. Then meet two different targets.
You may meet the ROC curve, which plots recall (the true positive rate) against the false positive rate at every possible threshold, and its summary number, the AUC (area under the curve): 0.5 means no better than a coin flip at ranking positives above negatives, and 1.0 means perfect separation. It compares models before you choose a threshold.
- ROC curve
- A plot of true positive rate against false positive rate across all thresholds.
- AUC
- Area under the ROC curve: 0.5 is random ranking, 1.0 is perfect.
- False positive rate
- FP ÷ (FP + TN): the share of real negatives wrongly flagged.
Remember: Lower threshold: more recall, less precision. Price each mistake and pick the cheapest threshold.
Practise it in Machine Learning Fundamentals
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Confusion matrix and threshold lab.
Every lesson is free: no ads, no lives and no paywall.