Skip to content
FORGENO ADS GAMESUnlock all $8

Responsible AI & AI Safety · Unit 2 of 8

Unit 2: Bias and fairness

Where bias comes from, and fairness metrics worked by hand: selection rates and the four-fifths rule, equal opportunity, equalized odds, calibration, and why they conflict.

Opens the first lesson of this unit in Responsible AI & AI Safety. Every lesson is open from the start.

Lesson 1: Where bias comes from

Goal: Trace bias to its source in the AI life cycle.

In AI, bias means a systematic error that falls unevenly on groups of people. It rarely comes from anyone intending harm. It enters at each stage of building a system, which is why removing a sensitive column such as race or sex almost never removes the bias.

SourceWhat happensExample
Historical biasThe world the data records was itself unfairPast hiring favored men, so "successful hire" labels favor men
Representation biasSome groups are missing or rare in the dataA skin-condition model trained mostly on light skin
Measurement biasA feature or label measures something different for different groupsArrests used as a label for crime, when some areas are policed more
Proxy featuresOther features stand in for a protected traitPostcode or first name standing in for race
Aggregation biasOne model for groups that differ in important waysA single diabetes model ignoring known differences between groups
Evaluation biasThe test data does not reflect the people the system will serveTesting a face model on a benchmark of mostly light-skinned men
Deployment biasThe system is used for a different purpose or populationA tool built to support staff becomes an automatic decision-maker

Published evidence shows how this plays out. The 2018 *Gender Shades* study (Buolamwini and Gebru) found commercial face-analysis systems misclassified the gender of darker-skinned women far more often, with error rates up to about 35%, than lighter-skinned men, under 1%. In December 2019, NIST's evaluation of 189 face recognition algorithms found many had higher false match rates for some demographic groups, sometimes by a factor of 10 to 100, though the most accurate algorithms showed much smaller differences.

Worked example: why deleting the "sex" column did not help

  1. A company trains a promotion model on ten years of past promotions. Few women were promoted, so the labels encode that history.
  2. It deletes the "sex" column to make the model "blind".
  3. The model still finds signals that correlate with sex: part-time years, career gaps, certain job titles, certain clubs on a CV.
  4. Women are still scored lower. The bias came from the labels and proxies, not the column.
  5. Fixes have to address the source: question the label, test outcomes by group, and remove or adjust proxies that carry unjustified differences.

"Fairness through unawareness" rarely works. You usually need the sensitive data to measure the problem.

Bias (in AI)
A systematic error that falls unevenly on groups of people.
Historical bias
Unfairness already present in the world that the data records.
Representation bias
Under- or over-representation of groups in training or test data.
Measurement bias
A feature or label that measures different things for different groups.
Protected characteristic
A trait that anti-discrimination law protects, such as race, sex, age, religion or disability.
Fairness through unawareness
Removing the sensitive attribute and hoping that makes a model fair. It usually fails because of proxies.

Remember: Bias enters through labels, data coverage, measurement, proxies, testing and deployment. Look at every stage.

Lesson 2: Selection rates and demographic parity

Goal: Compute selection rates, the impact ratio and the four-fifths rule by hand.

The simplest fairness check looks only at outcomes: what share of each group gets the good result? That share is the selection rate: number selected ÷ number in the group. Demographic parity (also called statistical parity) holds when selection rates are equal across groups.

In US employment, the 1978 Uniform Guidelines on Employee Selection Procedures introduced a rule of thumb, the four-fifths rule: if a group's selection rate is less than 80% of the highest group's rate, that is generally treated as evidence of adverse impact, which the employer may then need to justify. It is a screening signal, not a final legal test. New York City's Local Law 144 (enforced since July 5, 2023) requires employers using automated hiring tools to publish an independent bias audit with the same kind of number, called the impact ratio: a group's selection rate divided by the rate of the most-selected group.

Worked example: a bias audit by hand

  1. An invented screening tool sees 200 applicants from group X and advances 60. Selection rate: 60 ÷ 200 = 0.30, or 30%.
  2. It sees 150 applicants from group Y and advances 30. Selection rate: 30 ÷ 150 = 0.20, or 20%.
  3. The most-selected group is X at 30%.
  4. Impact ratio for Y: 0.20 ÷ 0.30 = 0.667.
  5. Four-fifths check: 0.667 is below 0.80, so this is a red flag for adverse impact on group Y.
  6. Demographic parity difference: 30% − 20% = 10 percentage points.

Selection rate per group, then divide by the highest. Below 0.8 means investigate.

Demographic parity is easy to measure and to explain, and it matches the idea that opportunities should be shared. Its weakness: it ignores whether people were actually qualified. If qualification rates truly differ between groups in the data, equal selection rates mean accepting less-qualified people in one group or rejecting more-qualified people in another. That may still be the right policy choice, for example when the "qualified" label itself is biased, but it is a choice that should be made openly.

Selection rate
The share of a group that receives the positive outcome: number selected ÷ number in the group.
Demographic parity
Equal selection rates across groups. Also called statistical parity.
Impact ratio
A group's selection rate divided by the selection rate of the most-selected group.
Four-fifths rule
A US employment rule of thumb: an impact ratio below 0.8 is generally treated as evidence of adverse impact.
Adverse impact
A substantially lower rate of selection for a group, which may indicate discrimination (also called disparate impact).
Bias audit
An evaluation of a system's outcomes by group, such as the audits required by New York City Local Law 144.

Remember: Selection rate = selected ÷ group size. Impact ratio = group rate ÷ highest rate. Below 0.8: investigate.

Lesson 3: Error rates, equal opportunity and calibration

Goal: Compute TPR, FPR and precision per group, and tell equal opportunity, equalized odds and calibration apart.

Error-rate fairness asks a sharper question: among people who really were qualified (or really would repay, or really were healthy), does the system treat each group alike? To answer, build a confusion matrix for each group: true positives (TP, qualified and selected), false positives (FP, not qualified but selected), false negatives (FN, qualified but rejected) and true negatives (TN).

MeasureFormulaQuestion it answers
True positive rate (TPR, recall)TP ÷ (TP + FN)Of the qualified people, what share were selected?
False positive rate (FPR)FP ÷ (FP + TN)Of the unqualified people, what share were wrongly selected?
Precision (positive predictive value, PPV)TP ÷ (TP + FP)Of the people selected, what share were really qualified?

Equal opportunity requires equal TPR across groups: qualified people have the same chance of being selected whatever their group. Equalized odds is stricter: equal TPR and equal FPR. Calibration is about scores: a score is calibrated within each group if, among people given a score of 0.7, about 70% really turn out positive, in every group. At the level of yes-or-no decisions, its cousin is predictive parity: equal precision across groups, so a "selected" label means the same thing for everyone.

Worked example: two confusion matrices

  1. Group X (invented): 100 applicants, 50 qualified. The tool selects 45: 40 qualified (TP) and 5 not (FP). So FN = 50 − 40 = 10 and TN = 50 − 5 = 45.
  2. Group X: TPR = 40 ÷ 50 = 0.80; FPR = 5 ÷ 50 = 0.10; precision = 40 ÷ 45 ≈ 0.89.
  3. Group Y: 100 applicants, 20 qualified. The tool selects 20: 12 qualified (TP) and 8 not (FP). So FN = 8 and TN = 80 − 8 = 72.
  4. Group Y: TPR = 12 ÷ 20 = 0.60; FPR = 8 ÷ 80 = 0.10; precision = 12 ÷ 20 = 0.60.
  5. FPRs are equal (0.10), but TPRs are not: qualified people in Y are selected 60% of the time versus 80% in X. Equal opportunity fails, so equalized odds fails too, and precision differs as well.

Always compute the matrix per group. One overall accuracy figure would hide all of this.

Which errors matter most depends on the setting. In hiring, a false negative denies a qualified person a job, so equal opportunity (equal TPR) is often the focus. In criminal justice, a false positive can mean someone is detained who would not have reoffended, so equal FPR gets attention. Naming the costly error is the first step in choosing a metric.

Confusion matrix
A table of true positives, false positives, false negatives and true negatives.
True positive rate (TPR)
TP ÷ (TP + FN): the share of real positives that the system catches. Also called recall.
False positive rate (FPR)
FP ÷ (FP + TN): the share of real negatives wrongly flagged as positive.
Precision (PPV)
TP ÷ (TP + FP): the share of positive decisions that were correct.
Equal opportunity
Equal true positive rates across groups.
Equalized odds
Equal true positive rates and equal false positive rates across groups.
Calibration
Scores mean the same thing in every group: among people scored 0.7, about 70% are positive.
Predictive parity
Equal precision across groups, so a positive decision is equally reliable for everyone.

Remember: TPR = TP/(TP+FN), FPR = FP/(FP+TN), precision = TP/(TP+FP). Equal opportunity: equal TPR. Equalized odds: equal TPR and FPR.

Lesson 4: Why fairness metrics conflict

Goal: Explain why fairness definitions cannot all hold at once, and choose one openly.

Here is the hard truth at the center of algorithmic fairness, proved independently by Kleinberg, Mullainathan and Raghavan (2016) and by Chouldechova (2017): when two groups have different base rates (different shares of real positives), no imperfect classifier can be calibrated, or have equal precision, and have equal false positive and false negative rates in both groups at the same time. Something has to give.

This is exactly the COMPAS dispute. ProPublica showed the false positive rates differed by race (an equalized-odds failure). The vendor replied that a given score meant the same reoffending risk for both groups (calibration and predictive parity). Both could be right at once, because reoffending base rates in the data differed. The real disagreement was about which kind of fairness should come first, which is a question of values and law, not of arithmetic.

Hands-on lab in the room: Fairness threshold lab. Move a separate threshold for each group, meet each fairness target on its own, then try for all of them at once.

Worked example: what the lab shows

  1. The lab's two groups have calibrated scores, but 61% of group A and 39% of group B were qualified.
  2. Same threshold, 0.5, for both groups: selection rates 70% (A) and 40% (B), an impact ratio of 0.57. TPRs are 85% and 67%. Scores are calibrated, yet outcomes differ sharply.
  3. Thresholds 0.6 (A) and 0.3 (B): selection rates are both 60%. Demographic parity holds, but precision is 78% versus 55%.
  4. Thresholds 0.5 (A) and 0.3 (B): TPRs are 85% and 85%, and FPRs 46% and 44%. Equalized odds roughly holds, but selection rates (70% and 60%) and precision (74% and 55%) differ.
  5. Search every pair of thresholds: none meets demographic parity, equalized odds and predictive parity together.

With different base rates, choosing a fairness measure is choosing which inequality you are willing to accept.

How professionals choose: (1) name the decision and who bears each kind of error; (2) check the law, since some regimes look at outcome rates and some forbid treating groups differently, including setting different thresholds by group; (3) question the label, because if "qualified" was measured with bias, matching error rates on it can lock that bias in; (4) consider fixing the data or process rather than the threshold; (5) document the choice, the trade-off and who approved it. Mitigations exist at each stage: pre-processing (fix or reweight the data), in-processing (train with a fairness constraint) and post-processing (adjust decisions after scoring).

Base rate
The share of real positives in a group, such as the share who are qualified.
Impossibility result
The proof that, with different base rates, calibration and equal error rates cannot all hold for an imperfect classifier.
Pre-processing
Reducing bias by changing the training data, for example by reweighting or fixing labels.
In-processing
Reducing bias during training, for example with a fairness constraint.
Post-processing
Reducing bias by adjusting decisions after the model scores, for example with thresholds.

Remember: Different base rates mean you cannot have calibration and equal error rates together. Choose openly, document why.

Practise it in Responsible AI & AI Safety

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Fairness threshold lab.

Every lesson is free: no ads, no lives and no paywall.