Skip to content
FORGENO ADS GAMESUnlock all $8

Machine Learning Fundamentals · Unit 5 of 8

Unit 5: Classification and its metrics

From scores to decisions with logistic regression and thresholds, the confusion matrix, why accuracy misleads, precision and recall, and choosing a threshold from real costs.

Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.

Lesson 1: From scores to classes

Goal: Explain classifier scores, thresholds and logistic regression.

Many classifiers do not output a label directly. They output a score, often a probability between 0 and 1: "this email is 0.92 likely to be spam". A threshold turns the score into a decision: with a threshold of 0.5, anything scoring 0.5 or more is treated as spam.

Logistic regression is the classic example. It computes a weighted sum of the features, just like linear regression, then passes it through the sigmoid function, which squashes any number into the range 0 to 1. Large positive sums become scores near 1, large negative sums scores near 0, and a sum of exactly 0 becomes 0.5.

The sigmoid turns any weighted sum into a score between 0 and 1
Weighted sumSigmoid score
−40.02
−10.27
00.50
10.73
40.98

Logistic regression is trained like linear regression, with gradient descent, but with a loss suited to probabilities (called log loss or cross-entropy) that punishes confident wrong answers heavily. The threshold, though, is not learned. It is a business decision, and the last lesson in this unit shows how moving it changes which mistakes you make.

Score (probability)
A classifier's number between 0 and 1 saying how strongly an example looks like the positive class.
Threshold
The score at or above which an example is labeled positive.
Logistic regression
A classifier that passes a weighted sum of features through the sigmoid to get a score.
Sigmoid
The S-shaped function that maps any number to a value between 0 and 1.

Remember: Weighted sum → sigmoid → score → threshold → decision. The threshold is your choice.

Lesson 2: The confusion matrix

Goal: Fill in a confusion matrix and calculate accuracy, and explain the accuracy paradox.

To evaluate a classifier, count its four kinds of outcome on the test set. Take a fraud model where "positive" means "flagged as fraud".

The confusion matrix
Actually fraudActually fine
Model says fraudTrue positive (TP): caughtFalse positive (FP): false alarm
Model says fineFalse negative (FN): missedTrue negative (TN): correctly ignored

Worked example: counting outcomes

1,000 test transactions, 20 of them really fraud. The model flags 30.

  1. Of the 30 flagged, 15 are fraud (TP = 15) and 15 are fine (FP = 15).
  2. Of the 20 frauds, 15 were caught, so 5 were missed (FN = 5).
  3. Everything else is a correct "fine": TN = 1,000 − 15 − 15 − 5 = 965.
  4. Accuracy = (TP + TN) ÷ total = (15 + 965) ÷ 1,000 = 98%.

98% sounds excellent, yet a quarter of the frauds got through.

This is the accuracy paradox: when one class is rare, accuracy is dominated by the easy majority. A model that flags nothing at all scores 980 ÷ 1,000 = 98% here too, and catches zero fraud. That is why professionals report precision and recall, the subject of the next lesson.

Confusion matrix
A table counting true positives, false positives, false negatives and true negatives.
True positive (TP)
A positive case correctly flagged.
False positive (FP)
A negative case wrongly flagged: a false alarm.
False negative (FN)
A positive case the model missed.
True negative (TN)
A negative case correctly left alone.
Accuracy
(TP + TN) ÷ all cases.

Remember: Count TP, FP, FN and TN before you trust any accuracy figure.

Lesson 3: Precision and recall

Goal: Calculate precision, recall and F1, and decide which matters in a situation.

Precision answers: when the model says positive, how often is it right? Precision = TP ÷ (TP + FP).

Recall answers: of all the real positives, how many did the model find? Recall = TP ÷ (TP + FN).

Worked example: the fraud model again

  1. TP = 15, FP = 15 and FN = 5.
  2. Precision = 15 ÷ (15 + 15) = 50%: half the flags are false alarms.
  3. Recall = 15 ÷ (15 + 5) = 75%: a quarter of the frauds slip through.
  4. The F1 score combines the two: 2 × P × R ÷ (P + R) = 2 × 0.5 × 0.75 ÷ 1.25 = 0.6.

Precision and recall tell you what kind of mistakes the model makes. Accuracy did not.

SituationWhich matters moreWhy
Spam filterPrecisionA real email lost in the spam folder is costly; a little spam in the inbox is not
First-stage screening for a serious disease, followed by a proper testRecallMissing a real case is far worse than an extra follow-up test
Fraud alerts that freeze cardsBoth, balancedMissed fraud costs money; false freezes anger customers
Recommending a filmPrecisionBad suggestions annoy; missing some good ones costs little
Precision
TP ÷ (TP + FP): how often a positive prediction is right.
Recall
TP ÷ (TP + FN): the share of real positives the model finds.
F1 score
A single number combining precision and recall: 2PR ÷ (P + R).

Remember: Precision: are my alarms right? Recall: did I catch them all? Pick by the cost of each mistake.

Lesson 4: Thresholds and trade-offs

Goal: Move a threshold to trade precision against recall, and choose it from real costs.

Moving the threshold trades precision against recall. Lower it and the model flags more: recall rises (it finds more real positives) and precision usually falls (more false alarms). Raise it and the opposite happens. There is no free lunch: you are choosing which mistakes to make more of.

Worked example: choosing a threshold from costs

A missed fraud costs about 500; a false alarm costs about 20 in staff time and customer annoyance.

  1. At threshold 0.7: FN = 10 and FP = 20, so cost = 10 × 500 + 20 × 20 = 5,400.
  2. At threshold 0.4: FN = 4 and FP = 90, so cost = 4 × 500 + 90 × 20 = 3,800.
  3. At threshold 0.2: FN = 2 and FP = 300, so cost = 2 × 500 + 300 × 20 = 7,000.
  4. The middle threshold costs least.

Put real costs on each kind of mistake and the choice of threshold becomes a calculation, not a guess.

Hands-on lab in the room: Confusion matrix and threshold lab. Slide the threshold on a scored test set and watch the confusion matrix, precision and recall move. Then meet two different targets.

You may meet the ROC curve, which plots recall (the true positive rate) against the false positive rate at every possible threshold, and its summary number, the AUC (area under the curve): 0.5 means no better than a coin flip at ranking positives above negatives, and 1.0 means perfect separation. It compares models before you choose a threshold.

ROC curve
A plot of true positive rate against false positive rate across all thresholds.
AUC
Area under the ROC curve: 0.5 is random ranking, 1.0 is perfect.
False positive rate
FP ÷ (FP + TN): the share of real negatives wrongly flagged.

Remember: Lower threshold: more recall, less precision. Price each mistake and pick the cheapest threshold.

Practise it in Machine Learning Fundamentals

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Confusion matrix and threshold lab.

Every lesson is free: no ads, no lives and no paywall.