Skip to content
FORGENO ADS GAMESUnlock all $8

Machine Learning Fundamentals · Unit 7 of 8

Unit 7: Decision trees

How a tree of yes-or-no questions makes predictions, choosing splits with Gini impurity, controlling depth to avoid overfitting, and random forests and boosting.

Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.

Lesson 1: How a decision tree decides

Goal: Follow a decision tree to a prediction and explain why trees are easy to explain.

A decision tree predicts by asking a series of yes-or-no questions about the features, like a flowchart. Each question is a split (a node), and each final answer is a leaf. To predict, start at the top, called the root, and follow the answers down.

Is monthly usage below 3 hours?
├─ yes → Has the customer contacted support in the last month?
│        ├─ yes → CANCEL  (14 of 18 training customers here cancelled)
│        └─ no  → STAY    (31 of 40 stayed)
└─ no  → STAY  (180 of 195 stayed)

Trees are popular in business because they are easy to explain: you can show a manager the exact path behind a prediction. They handle numbers and categories without scaling, and they capture interactions, such as "low usage matters more for customers who have contacted support", that a single straight line cannot.

A leaf predicts the majority class of the training examples that ended up in it (for a regression tree, their average). The counts in each leaf also give a rough confidence: 14 of 18 is about 78%.

Decision tree
A model that predicts by following a path of yes-or-no questions about the features.
Split (node)
One question in a tree, such as "usage below 3 hours?".
Leaf
An end point of a tree, holding a prediction.
Root
The first split at the top of a tree.

Remember: Follow the questions from the root to a leaf. The leaf's majority is the prediction.

Lesson 2: Choosing the best split

Goal: Calculate Gini impurity and compare splits.

A tree is built greedily: at each node it tries every feature and threshold and keeps the split that makes the resulting groups purest, each group as close as possible to a single class.

A common measure is Gini impurity: Gini = 1 − (the sum of the squared class proportions). A pure group, all one class, has Gini 0. A 50/50 group of two classes has Gini 0.5, the worst case for two classes.

Worked example: scoring a split

10 customers: 5 cancel and 5 stay, so Gini = 1 − (0.5² + 0.5²) = 0.5. Candidate split: usage below 3 hours?

  1. Yes group: 4 customers, all 4 cancel. Gini = 1 − (1² + 0²) = 0.
  2. No group: 6 customers, 1 cancels and 5 stay. Gini = 1 − ((1/6)² + (5/6)²) = 1 − (0.028 + 0.694) ≈ 0.28.
  3. Weighted Gini after the split = (4/10) × 0 + (6/10) × 0.28 ≈ 0.17.
  4. The split cuts impurity from 0.5 to 0.17. The tree compares this with every other candidate split and keeps the best.

Splitting is just "which question best separates the classes?", asked again and again.

Hands-on lab in the room: Decision tree builder. Choose the questions for a small tree yourself, see the Gini impurity of each split, and compare training and test accuracy. Then let the tree grow without limits.

Another common measure is entropy, used to compute information gain. It ranks splits very much like Gini does; the idea, purer groups, is the same.

Gini impurity
1 minus the sum of squared class proportions: 0 for a pure group, higher for mixed ones.
Purity
How close a group is to containing only one class.
Information gain
How much a split reduces entropy, an alternative to Gini for choosing splits.
Greedy algorithm
A method that makes the best choice at each step without looking ahead.

Remember: Gini = 1 − Σ(proportion²). Choose the split with the lowest weighted Gini.

Lesson 3: Depth and overfitting

Goal: Control tree overfitting with depth limits, leaf sizes and pruning.

Left alone, a tree keeps splitting until every leaf is pure, often with a single training example per leaf. That is memorization: perfect training accuracy and disappointing test accuracy.

  • Maximum depth: limit how many questions can be asked in a row, for example 4.
  • Minimum samples per leaf: never create a leaf with fewer than, say, 20 examples.
  • Pruning: grow a large tree, then cut back branches that do not improve validation accuracy.
A typical pattern: depth 3 generalizes best here.
Maximum depthTraining accuracyTest accuracy
172%71%
384%82%
693%80%
Unlimited100%69%

Worked example: reading a deep leaf

  1. A depth-12 tree has a leaf: "usage below 2.7 hours, joined in March, lives in a postcode starting HU, two support contacts: CANCEL (1 of 1)".
  2. One customer is not a pattern. The leaf has memorized an individual.
  3. With a minimum of 20 samples per leaf, this branch would never have been created.

Leaves built from a handful of examples are memories, not patterns.

Maximum depth
The most questions a tree may ask in a row.
Pruning
Removing branches of a tree that do not improve validation performance.
Minimum samples per leaf
The smallest number of training examples a leaf may hold.

Remember: Limit depth, require enough examples per leaf, or prune. Tune these on validation data.

Lesson 4: Forests and boosting

Goal: Explain random forests and gradient boosting, and read feature importance carefully.

Single trees are unstable: change a few training examples and the whole tree can change. Ensembles combine many trees to fix that.

  • Random forest: train hundreds of trees, each on a random sample of the rows and considering a random subset of features at each split. To predict, they vote (or average). The randomness makes the trees different, so their errors partly cancel out.
  • Gradient boosting: train trees one after another, each new tree focusing on the errors the earlier trees still make. Boosted trees are among the most accurate methods for tabular business data, the kind found in spreadsheets and databases.
ModelStrengthsWeaknesses
Single treeEasy to explainUnstable, overfits easily
Random forestAccurate, robust, little tuningHarder to explain, larger
Gradient boostingOften the most accurate on tablesNeeds careful tuning, can overfit

Worked example: feature importance, read with care

  1. A forest reports feature importance: usage 0.41, support contacts 0.27, contract type 0.19, age 0.13.
  2. That tells you which features the forest relied on most to separate the classes.
  3. It does not tell you that changing usage will change cancellations: importance is association, not cause.
  4. Importance can also be shared between features that carry the same information, making each look less important than it really is.

Feature importance helps you understand a model. It is not a substitute for an experiment.

Ensemble
A model made by combining many models.
Random forest
An ensemble of trees trained on random samples of rows and features, which vote.
Gradient boosting
An ensemble that adds trees one at a time, each correcting the errors of the earlier ones.
Feature importance
A measure of how much a model relied on each feature.

Remember: Forests average many varied trees; boosting adds trees that fix earlier errors. Importance is not cause.

Practise it in Machine Learning Fundamentals

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Decision tree builder.

Every lesson is free: no ads, no lives and no paywall.