AI Data Annotation & Model Rating · Unit 5 of 8
Unit 5: Rating AI responses
The four things raters judge, 1 to 7 scales and calibration, side-by-side preferences, and how your comparisons train a model through RLHF.
Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.
Lesson 1: The four things raters judge
Goal: Rate a response on instruction-following, helpfulness, honesty and accuracy, and harmlessness, and weigh how serious each problem is.
Most AI trainer projects ask you to judge a chatbot's response to a prompt. Every project words its rubric differently, but almost all of them check the same four things.
| Dimension | The question it asks | A typical failure |
|---|---|---|
| Instruction-following | Did it do what the prompt asked, within every constraint: length, format, language, audience, things to include or avoid? | Asked for three bullet points, gave five paragraphs |
| Helpfulness | Does it actually solve the person's problem, at the right depth, clearly and without padding? | Restates the question, then gives generic advice |
| Honesty and accuracy | Are its facts, numbers, code and reasoning correct, and is it open about uncertainty and its limits? | Invents a statistic or a source; states a guess as fact |
| Harmlessness | Does it avoid giving real help with harm, and avoid needless refusals, lectures or unsafe advice? | Gives dangerous instructions, or refuses a harmless request |
Read the prompt before the response. List every explicit constraint and what a good answer would need, then check the response against your list. Raters who read the response first tend to be swayed by how confident and polished it sounds, and polish is the easiest thing for a model to fake.
Severity matters more than count. One false fact in a health, legal or money answer can make a polished response worse than a plain correct one, and many guidelines say a serious accuracy or safety problem caps the overall score however well the rest is written. Small issues, such as a slightly stiff tone, lower a score a little; they do not sink it. And "harmless" does not mean "refuse when unsure": an unnecessary refusal of a safe request is a helpfulness failure, and projects mark it down.
Worked example: rating one response
Prompt: "In under 100 words, explain to a 10-year-old why the sky is blue." The response runs to 160 words. It says sunlight contains every colour, that air scatters blue light more than red, and that "the sky is blue because it reflects the ocean".
- Constraints from the prompt: under 100 words; suitable for a 10-year-old; explain why.
- Instruction-following: 160 words breaks the limit. The language is simple enough for a child.
- Honesty and accuracy: the scattering part is right, but "it reflects the ocean" is a common myth, so the response contains a false fact.
- Helpfulness: a child would get a mostly good explanation plus a myth that contradicts it.
- Harmlessness: no problem.
Two real problems, one of them a false fact: a middling-to-low score, with a note that names both. "Over the 100-word limit (160 words); repeats the myth that the sky reflects the ocean."
- Instruction-following
- Doing exactly what the prompt asked, within every stated constraint such as length, format, language and audience.
- Helpfulness
- How well a response actually solves the person's problem, at the right depth and without padding.
- Honesty (AI responses)
- Being accurate and open about uncertainty: no invented facts, sources or abilities.
- Harmlessness
- Avoiding real help with harm, unsafe advice and needless refusals or lectures.
- Over-refusal
- Refusing or hedging on a request that is safe to help with. It counts as a helpfulness failure.
- Hallucination
- A fluent but false statement produced by a model, such as an invented fact, quote or source.
Remember: Read the prompt first and list its constraints. A serious accuracy or safety problem outweighs any amount of polish.
Lesson 2: Rating scales and calibration
Goal: Use a 1 to 7 scale with anchors, spot leniency, severity, central tendency and halo effects, and justify every score.
Many projects ask for a score on a scale, often 1 to 5 or 1 to 7, for each dimension and for the response overall. A number only means something if everyone uses it the same way, so good rubrics give anchors: a short description of what each score looks like. The table is an example; on a real project always use that project's own anchors.
| Score | Example anchor |
|---|---|
| 7 Excellent | Fully correct, follows every instruction, could not reasonably be improved. Rare: never give it out of habit. |
| 6 Very good | Correct and complete, with one or two very minor issues. |
| 5 Good | Correct, with small gaps or style issues a user would barely notice. |
| 4 Acceptable | Useful, but with a noticeable gap, such as a missed minor constraint. |
| 3 Weak | A real problem, such as a missed main constraint or a minor factual error. |
| 2 Poor | Mostly unhelpful, or a significant factual error. |
| 1 Unacceptable | Harmful, completely wrong, or does not attempt the task. |
Calibration means your scores match the project's experts on the same items. The common biases have names. Leniency: everything gets 6s. Severity: everything gets 3s. Central tendency: hiding in the middle with 4s whenever a call feels hard. The halo effect: one strong quality, such as confident, fluent writing, lifts every other score. Projects find these by comparing your scores with gold scores. Your average difference from them, and whether the differences all point the same way, shows your bias.
Every score needs a justification that points to evidence: name or quote the problem, and say which dimension it affects and how much. "Good answer" is not a justification. "Correct and complete, but 140 words against a 100-word limit, so instruction-following 4" is, and a reviewer can check it in seconds.
Worked example: checking your own calibration
- You rate five responses for accuracy: 6, 7, 5, 6, 7. The expert scores are 4, 6, 4, 5, 6.
- Your differences (yours minus the expert's): +2, +1, +1, +1, +1.
- Average difference = 6 ÷ 5 = +1.2, and every difference is positive.
- That is leniency: about one point too generous, every time. You reread the anchors for 4 and 5, and start writing down the specific issue before you choose a number.
Differences that all share a sign are a bias you can fix. Differences scattered both ways are ordinary noise.
Hands-on lab in the room: Rubric rating and calibration lab. Rate three responses from 1 to 7 on three dimensions. The lab compares every score with an expert's, explains the expert's reason, and shows whether you lean lenient or harsh.
- Likert scale
- A rating scale with ordered, labelled points, such as 1 to 7.
- Anchor
- A description of what a particular score on a rating scale looks like.
- Calibration (rating)
- Scoring the way the project's experts score the same items.
- Leniency bias
- Scoring consistently higher than the standard.
- Severity bias
- Scoring consistently lower than the standard.
- Central tendency bias
- Avoiding the ends of a scale and giving middle scores whenever a call is hard.
- Halo effect
- Letting one strong quality, such as fluent writing, raise every other score.
- Justification (rationale)
- The written evidence behind a score or preference.
Remember: Anchor every score, justify it with evidence, and check whether your differences from gold all share a sign.
Lesson 3: Side-by-side preference ranking
Goal: Compare two responses, choose a preference and its strength, avoid length bias and sycophancy, and write a rationale that leads with the deciding reason.
In a side-by-side (SxS) or pairwise preference task you see one prompt and two responses, A and B, and say which is better and by how much. A common scale runs: A much better · A better · A slightly better · About the same · B slightly better · B better · B much better. Many projects also ask you to score each response on its own first, and to write a short rationale.
- Read the prompt and list its constraints and what an ideal answer would contain.
- Check each response alone for deal-breakers: false facts, safety problems, ignored instructions.
- Compare on what matters most. A deal-breaker in one response usually decides the pair. If neither has one, compare helpfulness, completeness and clarity.
- Choose the strength. "Much better" when one has a serious problem the other avoids; "slightly better" for differences of polish or tone.
- Write the rationale, deciding reason first, with evidence.
Watch for length bias. Longer answers look more thorough, and raters tend to reward them, so models trained on those choices learn to pad. A longer answer is better only if the extra length gives the person something they needed. Watch too for sycophancy: a response that goes along with a false belief in the prompt is less honest than one that politely corrects it and then helps. "About the same" is a real answer when two responses really are equally good or equally flawed, but using it to dodge a hard call is a quality problem in its own right.
Worked example: one pair
Prompt: "In two sentences, why do bees matter to farming?" Response A: four accurate sentences about pollination and honey. Response B: "Bees pollinate many crops, such as apples and almonds, which helps the plants produce fruit and seeds. Without them, many farms would get smaller harvests or need costlier ways to pollinate."
- Constraint: two sentences. A breaks it; B keeps it.
- Both are accurate. A's extra material on honey is not what the question was about: bees matter to farming mainly through pollination.
- Neither has a safety problem.
- Preference: B better. Not "much better", because A has no serious error, only a missed constraint and some padding.
Rationale: "B keeps to the two-sentence limit and explains pollination clearly; A is accurate but runs to four sentences and adds less relevant detail about honey."
Hands-on lab in the room: Side-by-side response rater. Five prompts, each with two responses. Choose your preference, its strength and the main reason, write a rationale, then compare with an expert's judgment.
- Side-by-side (SxS) rating
- Comparing two responses to the same prompt and saying which is better, and by how much.
- Preference strength
- How much better one response is, from "slightly" to "much".
- Deal-breaker
- A problem serious enough to decide a comparison on its own, such as a false fact in an important answer or unsafe advice.
- Length bias
- Rewarding longer answers because they look thorough, even when the extra length adds nothing.
- Sycophancy
- A model telling people what they want to hear, such as agreeing with a false belief, instead of what is true.
Remember: Deal-breakers first, then helpfulness. Longer is better only when the length helps. Lead the rationale with the deciding reason.
Lesson 4: How your preferences train a model
Goal: Explain simply how RLHF-style training uses comparisons, count the pairs in a ranking, and explain reward hacking.
Why do companies pay people to compare answers? Because comparisons are a practical way to teach a model what "better" means. Even when people disagree about what score an answer deserves, they can often agree on which of two answers is better. The best-known recipe is called reinforcement learning from human feedback (RLHF). Simply put, it works like this:
- Collect comparisons. People like you see a prompt and two or more responses, and say which is better.
- Train a reward model. A separate model learns to predict which response people would prefer. It turns all those judgments into a score it can give any new response.
- Improve the assistant. The assistant is trained to produce responses the reward model scores highly, while being held back from drifting too far from how it behaved before.
- Repeat. New comparisons on the improved assistant catch new problems.
Newer methods, such as direct preference optimisation (DPO), skip the separate reward model and learn from the preference pairs directly. They still need exactly the same thing from you: careful, consistent comparisons. A ranking is turned into pairs. If you rank four responses A > B > C > D, that gives six pairs: A>B, A>C, A>D, B>C, B>D and C>D. In general, a ranking of n responses gives n × (n − 1) ÷ 2 pairs, and each pair is one training signal.
This is why rater habits matter so much. If raters keep preferring longer answers, the reward model learns "longer is better", and the assistant learns to pad. If raters reward a confident tone over correctness, the assistant learns to sound sure when it is not. A model that scores well by exploiting such blind spots, rather than by actually being better, is said to be reward hacking, and careful raters are one of the main defences against it. Teams also read rationales to audit the data and to find gaps in the guideline.
Worked example: from a ranking to training pairs
- You rank three responses to one prompt: B best, then A, then C.
- That gives three pairs: B over A, B over C, and A over C.
- The reward model is trained so that, for this prompt, B scores above A and A scores above C.
- Suppose C was the longest response, and many raters on the project ranked long answers first out of habit. The reward model would learn to favour length and the assistant would learn to pad. Your ranking pushes the other way, but only because you judged on substance.
Every comparison is a small vote on what the model will become. Consistent, guideline-based votes make a better model.
- Reinforcement learning from human feedback (RLHF)
- Training an assistant using people's comparisons of its responses, usually through a reward model.
- Reward model
- A model trained to predict which response people would prefer, used to score new responses.
- Preference pair
- One judgment that response X is better than response Y for the same prompt.
- Direct preference optimisation (DPO)
- A method that trains a model directly on preference pairs, without a separate reward model.
- Reward hacking
- A model scoring well by exploiting blind spots in what is rewarded, such as length, instead of actually improving.
Remember: A ranking of n responses gives n × (n − 1) ÷ 2 pairs. Each one teaches the model what "better" means, so judge substance, not length or confidence.
Practise it in AI Data Annotation & Model Rating
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Rubric rating and calibration lab and Side-by-side response rater.
Every lesson is free: no ads, no lives and no paywall.