Skip to content
FORGENO ADS GAMESUnlock all $8

Machine Learning Fundamentals · Unit 4 of 8

Unit 4: Linear regression and gradient descent

Predicting numbers with a line, measuring error with MSE, how gradient descent finds the best line step by step, learning rates, and models with several features.

Opens the first lesson of this unit in Machine Learning Fundamentals. Every lesson is open from the start.

Lesson 1: Predicting a number with a line

Goal: Read and use a linear model ŷ = w × x + b.

Linear regression predicts a number with a straight line: ŷ = w × x + b. Here x is the feature, ŷ ("y-hat") is the prediction, w is the weight or slope (how much the prediction changes when x goes up by one) and b is the bias or intercept (the prediction when x is 0).

Worked example: delivery times

A courier firm finds that delivery time ≈ 4 × distance + 10, in minutes and kilometers.

  1. w = 4: each extra kilometer adds about 4 minutes.
  2. b = 10: even a delivery next door takes about 10 minutes, for loading, parking and stairs.
  3. A 6 km delivery: 4 × 6 + 10 = 34 minutes.
  4. A 0.5 km delivery: 4 × 0.5 + 10 = 12 minutes.

The two learned numbers, w and b, are the whole model, and you can read them in plain words.

Training means finding the w and b that make the line fit the training examples as closely as possible. The next lesson defines "as closely as possible"; the one after shows how a computer finds it.

Watch out: A line only predicts well inside the range of the data it was fitted on. If every training delivery was under 15 km, do not trust the model at 120 km: other factors, like motorways or a change of driver, take over. Predicting far outside the data is called extrapolation.

Linear regression
Predicting a number as a weighted sum of features plus a bias: for one feature, a straight line.
Weight
A learned number that says how much the prediction changes per unit of a feature.
Bias (intercept)
The learned constant added to the weighted sum: the prediction when every feature is 0.
Extrapolation
Predicting for inputs far outside the range of the training data.

Remember: ŷ = w × x + b. Read w as "per unit of x", b as the starting value. Stay inside the data's range.

Lesson 2: Measuring error: the loss

Goal: Compute residuals, MSE and MAE, and explain why errors are squared.

For each training example, the residual (error) is the actual value minus the prediction. To score a whole line, combine the errors. The usual choice for regression is mean squared error (MSE): square each error, then take the average.

Worked example: MSE by hand

Line: ŷ = 4x + 10. Three deliveries: (2 km, 20 min), (5 km, 28 min), (8 km, 45 min).

  1. Predictions: 18, 30 and 42 minutes.
  2. Errors (actual − predicted): 2, −2 and 3.
  3. Squared errors: 4, 4 and 9.
  4. MSE = (4 + 4 + 9) ÷ 3 ≈ 5.67.

A lower MSE means a better fit to these points. Training searches for the w and b with the lowest MSE.

Why square? Squaring makes every error positive, so misses above and below cannot cancel out, and it punishes big misses far more than small ones: one error of 10 costs as much as 100 errors of 1. That is often what you want, but it also makes MSE sensitive to outliers. Mean absolute error (MAE), the average of the absolute errors, treats every minute of error equally and is easier to explain: "on average we are 2.3 minutes off".

Whatever quantity training tries to make small is called the loss (or cost) function. Different tasks use different losses, but the idea is the same everywhere in machine learning, including the huge networks behind language models.

Residual
Actual value minus predicted value for one example.
Mean squared error (MSE)
The average of the squared residuals.
Mean absolute error (MAE)
The average of the absolute residuals.
Loss function
The measure of error that training tries to minimize.

Remember: Errors are actual minus predicted. MSE squares then averages; MAE averages the sizes.

Lesson 3: Gradient descent

Goal: Describe gradient descent and calculate one update step.

How does a computer find the best w and b? For a single straight line there is a formula, but most models, including neural networks, are trained with a general method called gradient descent.

Picture the loss as a landscape: every possible pair (w, b) is a location, and the height is the MSE. You start somewhere and want to reach the lowest valley, in fog. The gradient tells you which way is uphill and how steep it is. Gradient descent takes a small step the opposite way, recalculates, and repeats.

  1. Start with a guess, such as w = 0 and b = 0.
  2. Compute the predictions and the loss on the training data.
  3. Compute the gradient: how much the loss would change if w or b were nudged up.
  4. Update: w ← w − learning rate × (gradient for w), and the same for b.
  5. Repeat until the loss stops falling.

Worked example: one step, by hand

Data: (1, 3), (2, 5) and (3, 7), which fit y = 2x + 1 exactly. Start at w = 0 and b = 0, with a learning rate of 0.1.

  1. Predictions are all 0, so the errors (prediction − actual) are −3, −5 and −7. The MSE is (9 + 25 + 49) ÷ 3 ≈ 27.7.
  2. Gradient for w = 2 × average(error × x) = 2 × (−3 − 10 − 21) ÷ 3 ≈ −22.7. Gradient for b = 2 × average(error) = 2 × (−15 ÷ 3) = −10.
  3. Update: w = 0 − 0.1 × (−22.7) ≈ 2.27, and b = 0 − 0.1 × (−10) = 1.0.
  4. The MSE falls from about 27.7 to about 0.33 in this one step. Repeating the step brings w close to 2 and b close to 1.

Each step uses the gradient to move downhill. That is all training is, repeated many times.

Hands-on lab in the room: Gradient descent line fit. Fit a line by hand with sliders, then let gradient descent do it, and try different learning rates.

Gradient
The direction and steepness of the fastest increase in the loss, one number per parameter.
Gradient descent
Repeatedly nudging the parameters against the gradient to reduce the loss.
Learning rate
How big a step gradient descent takes each time.

Remember: New value = old value − learning rate × gradient. Repeat until the loss stops falling.

Lesson 4: Learning rates and more features

Goal: Choose a learning rate from the loss curve and read the weights of a model with several features.

The learning rate sets the step size. Too small, and training crawls along in thousands of tiny steps. Too large, and each step overshoots the valley and lands higher up the other side: the loss bounces about or grows without limit, called divergence. People choose it by trying values and watching the loss curve.

Learning rateWhat the loss curve does
Too smallFalls very slowly
About rightFalls quickly, then levels off (convergence)
Too largeZig-zags, or rises without limit (divergence)

Multiple linear regression uses several features, each with its own weight: price ≈ w₁ × size + w₂ × distance + w₃ × garden + b. Gradient descent works the same way, with one gradient per weight. Scaling the features first helps a lot: when one feature is in thousands and another in single digits, no single learning rate suits both.

Worked example: reading weights carefully

  1. A house-price model learns: price ≈ 2,100 × size_m² − 4,500 × distance_km + 8,000 × garden + 40,000.
  2. Each extra square meter adds about 2,100 to the predicted price, holding the other features fixed.
  3. Each kilometer from the center lowers it by about 4,500.
  4. Careful: weights show association in this data, not cause. And if two features move together, like size and number of bedrooms, their weights can share the credit in odd ways.

Weights make linear models explainable, but an explanation is not proof of cause.

Multiple linear regression
Linear regression with several features, each with its own weight.
Convergence
When the loss stops falling because training has reached a minimum.
Divergence
When steps are too large and the loss grows instead of falling.

Remember: Watch the loss curve: slow means raise the rate, exploding means lower it. Weights are associations, not causes.

Practise it in Machine Learning Fundamentals

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Gradient descent line fit.

Every lesson is free: no ads, no lives and no paywall.