AI Data Annotation & Model Rating · Unit 4 of 8
Unit 4: Consistency and agreement
Gold questions, audits and feedback; percent agreement and why it can mislead; Cohen's kappa worked by hand; and how teams raise agreement.
Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.
Lesson 1: Gold questions, audits and feedback
Goal: Explain how gold questions and audits measure your work, and respond to feedback well.
Projects need to know whether each person is reliable without checking every label. These are the main tools:
| Tool | How it works | What it measures |
|---|---|---|
| Gold questions (honeypots) | Items with a known right answer are hidden among normal tasks | Your accuracy, without you knowing which items are tests |
| Qualification test | A graded set of tasks before you join a project | Whether you understood the guideline |
| Audits or review sampling | A reviewer checks a random sample of your work | Accuracy, and the kinds of mistakes you make |
| Overlap | Several people label the same items | Agreement with others (Lessons 2 to 4) |
Gold questions usually look exactly like other tasks, and they tend to sit on the edge cases the guideline covers. So the best strategy is simply to treat every item as if it were gold. Projects often set a minimum gold accuracy, and falling below it can mean retraining, fewer tasks or removal from the project.
Feedback is data about you. When a reviewer marks something wrong, find the rule they used, decide whether it is a one-off slip or a pattern (Unit 1), and update your cheat sheet. If you believe a gold answer really is wrong, many projects have a dispute process: quote the rule, explain calmly and accept the outcome. Gold answers are sometimes wrong, and good disputes improve the project for everyone.
Worked example: reading your gold score
- This week you answered 50 gold questions without knowing which ones they were, and got 46 right.
- Gold accuracy = 46 ÷ 50 = 92%. Suppose the project minimum is 90%.
- All four misses were sarcastic messages. That is a pattern, not bad luck.
- You reread the sarcasm rule and add a cheat-sheet line. You dispute none of them, because the rule was clear.
Above the line, but with a clear pattern to fix before it costs you.
- Gold question (honeypot)
- A hidden test item with a known right answer, mixed in with normal tasks.
- Qualification test
- A graded set of tasks you must pass before joining a project.
- Audit
- A reviewer's check of a sample of your work.
- Dispute
- A formal, polite challenge to a gold answer or review that you believe is wrong.
Remember: Treat every item as gold, read feedback for patterns, and dispute calmly by quoting the rule.
Lesson 2: Percent agreement and its trap
Goal: Calculate percent agreement and explain why it can mislead.
When two people label the same items, the simplest measure is percent agreement: the share of items where they gave the same label. If two raters label 10 messages and give the same label on 8, they agree 80% of the time.
Percent agreement is easy to understand, but it has a trap: some agreement happens by chance. Imagine two raters who never read the comments and simply label almost everything NOT TOXIC. Because nearly every item gets the same label, they agree most of the time by luck alone. On data where one label dominates, a high percent agreement can hide the fact that raters disagree on exactly the rare items that matter most.
Worked example: agreement that means less than it looks
Two raters label 100 comments TOXIC or NOT TOXIC.
- Both say NOT TOXIC on 95 comments, and both say TOXIC on 1.
- On the other 4 they disagree: 2 that A called TOXIC and B did not, and 2 the other way round.
- Percent agreement = (95 + 1) ÷ 100 = 96%.
- But of the 5 comments that either rater thought might be toxic, they agreed on only 1.
96% sounds excellent, yet on the rare, important label they barely agree. Lesson 3 shows the measure that catches this.
| B: TOXIC | B: NOT TOXIC | Total for A | |
|---|---|---|---|
| A: TOXIC | 1 | 2 | 3 |
| A: NOT TOXIC | 2 | 95 | 97 |
| Total for B | 3 | 97 | 100 |
A table like this, with one rater down the side and the other across the top, is called an agreement table or confusion matrix. The cells on the diagonal are agreements; every other cell is a disagreement. It is the starting point for kappa, and it shows *which* labels people confuse.
- Percent agreement
- The share of items on which two raters gave the same label.
- Chance agreement
- Agreement that would happen even if raters labelled at random, using their usual label frequencies.
- Agreement table (confusion matrix)
- A grid counting how often each of one rater's labels met each of the other rater's labels. The diagonal holds the agreements.
- Class imbalance
- When one label is far more common than the others.
Remember: Percent agreement = same labels ÷ items. When one label dominates, it flatters.
Lesson 3: Cohen's kappa, by hand
Goal: Calculate Cohen's kappa step by step from an agreement table.
Cohen's kappa (κ) measures how much two raters agree beyond what chance would give. It compares the observed agreement, p_o, with the agreement you would expect by chance, p_e, which comes from how often each rater uses each label.
kappa = (p_o − p_e) ÷ (1 − p_e)
p_o = agreements ÷ total items
p_e = for each label, (share of items A gave it × share B gave it), all added up
Read the result like this: 1 means perfect agreement, 0 means no better than chance, and below 0 means worse than chance, which often means someone misread a definition. In between is a sliding scale. A commonly quoted rough guide calls about 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and above 0.80 almost perfect. Treat those bands as a rule of thumb, not a law: what counts as good depends on how hard the task is.
Worked example: kappa for a spam task
Two raters label 20 emails SPAM or NOT.
- From the agreement table: both SPAM 6, both NOT 8, A SPAM but B NOT 2, A NOT but B SPAM 4.
- p_o = (6 + 8) ÷ 20 = 0.70.
- Rater A said SPAM 8 times (0.4) and NOT 12 times (0.6). Rater B said SPAM 10 times (0.5) and NOT 10 times (0.5).
- p_e = 0.4 × 0.5 + 0.6 × 0.5 = 0.20 + 0.30 = 0.50.
- κ = (0.70 − 0.50) ÷ (1 − 0.50) = 0.20 ÷ 0.50 = 0.40.
70% agreement, but a kappa of 0.40: only fair to moderate once chance is taken out.
Now the toxic comments from Lesson 2. p_o = 0.96. Each rater said TOXIC 3 times in 100 (0.03) and NOT TOXIC 97 times (0.97), so p_e = 0.03 × 0.03 + 0.97 × 0.97 = 0.0009 + 0.9409 = 0.9418. κ = (0.96 − 0.9418) ÷ (1 − 0.9418) ≈ 0.31. Most of that 96% was luck.
Hands-on lab in the room: Agreement and kappa calculator. Label 16 messages yourself, then work out kappa between you and a second rater by hand before the lab shows its working.
- Cohen's kappa
- A measure of agreement between two raters beyond chance: (p_o − p_e) ÷ (1 − p_e).
- Observed agreement (p_o)
- The share of items where the two raters gave the same label.
- Expected agreement (p_e)
- The agreement chance alone would give, worked out from each rater's label shares.
- Inter-annotator agreement (IAA)
- Any measure of how consistently different annotators label the same items.
Remember: κ = (p_o − p_e) ÷ (1 − p_e). Get p_e by multiplying the two raters' shares for each label and adding.
Lesson 4: Raising agreement
Goal: Use calibration, adjudication and disagreement analysis, and tell a hard task from a bad rater.
Low agreement is a signal, and the first question is why. Look at the agreement table. If most disagreements sit in one cell, say A calls things MIXED that B calls POSITIVE, the problem is probably one unclear definition, not general carelessness.
| Cause | Sign in the data | Fix |
|---|---|---|
| Unclear definition | Disagreements cluster on one pair of labels | Rewrite the definition and add examples |
| Missing edge-case ruling | Many people disagree on the same few items | Add a ruling to the FAQ |
| One rater misreading | One person disagrees with everyone else, in the same direction | Targeted feedback or retraining |
| Genuinely subjective task | Agreement stays modest even after fixes | Collect several labels per item and keep the spread |
Calibration sessions bring annotators together, often on a call or in a shared document, to label the same small set, compare answers and discuss the rule behind each one. Adjudication means an expert reviews the items where annotators disagreed and decides the final label, which can then become a new example in the guideline. Both turn disagreements into clearer rules.
Some tasks are subjective by design, such as "How funny is this joke?" or "How natural does this voice sound?". There, a modest kappa is not a failure: the variety of opinion is real information, and projects often keep several ratings per item instead of forcing one answer.
Worked example: fixing a cluster
- Six annotators label 200 reviews. Kappa is 0.52.
- The agreement tables show that 70% of the disagreements are MIXED versus POSITIVE, on reviews with mild complaints ("a bit slow, but delicious").
- The lead adds a rule: "A mild complaint ('a bit', 'slightly') still counts as a clear negative opinion," with three examples.
- After a calibration session on 20 items, kappa on the next batch is 0.74.
One clarified definition did more than asking everyone to "be more careful".
- Calibration session
- A meeting where annotators label the same items, compare answers and agree on the rule behind each.
- Adjudication
- An expert deciding the final label for items where annotators disagreed.
- Subjective task
- A task where reasonable people can disagree, such as rating humour. The spread of opinion is information.
Remember: Find where disagreements cluster, fix the definition, calibrate and adjudicate. Some tasks are meant to vary.
Practise it in AI Data Annotation & Model Rating
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Agreement and kappa calculator.
Every lesson is free: no ads, no lives and no paywall.