AI Data Annotation & Model Rating · Unit 3 of 8
Unit 3: Guidelines, edge cases and escalation
How to read and apply a long guideline precisely, handle edge cases without guessing, and escalate the right things the right way.
Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.
Lesson 1: Reading a long guideline
Goal: Find your way around a long guideline and build a cheat sheet from it.
Real projects come with guidelines that run from a few pages to well over a hundred. Google's published Search Quality Rater Guidelines, for example, ran to about 180 pages in their 2025 versions. Nobody memorises a guideline on day one. The skill is knowing how guidelines are built, reading them in the right order, and keeping the rules that matter most within reach.
| Section | What it gives you | How to use it |
|---|---|---|
| Purpose and overview | What the data is for | Read first: it tells you what "good" means |
| Definitions | The exact meaning of each label | Read slowly. These decide most items |
| Decision rules | Steps or priorities for choosing a label | Follow them in order, especially "this rule beats that one" |
| Examples | Worked items with the right answer | Study the ones that look wrong to you |
| Edge cases and FAQ | Rulings on hard items | Search here before you escalate |
| Changelog | What changed, and when | Check it every time you start work |
Read in passes. First skim the whole document for its structure. Then read the purpose, definitions and decision rules closely. Then work through the examples, covering each answer and trying it yourself. Finally, build a cheat sheet: one page with the definitions in your own words, the priority order, and the rulings that surprised you. Keep it beside you while you work, and update it whenever the changelog changes.
The most important habit: the guideline beats your intuition. It may say that a review praising the food but complaining about the delivery is MIXED, or that a picture of a car is not a car. If you disagree with a rule, follow it anyway and raise your question through the proper channel (Lesson 4). Use your personal judgment only where the guideline asks for it.
Worked example: a cheat sheet entry
Across three pages the guideline says: "MIXED requires at least one clearly positive and one clearly negative opinion about the order or service. Opinions about other subjects (weather, other companies, the customer's day) are ignored. Sarcasm is labelled by intended meaning."
- Cheat-sheet line 1: MIXED = one clear + and one clear − about OUR order or service.
- Line 2: Ignore opinions about anything else.
- Line 3: Sarcasm = what they mean, not the words.
- Line 4, from the changelog for version 2.3: an emoji on its own counts as an opinion.
Four lines replace three pages for daily use, and the full guideline is still there for anything new.
- Guideline (annotation)
- The project rulebook: purpose, label definitions, decision rules, examples and edge cases.
- Changelog
- The list of guideline changes, with dates or version numbers.
- Cheat sheet
- Your one-page summary of the definitions, priorities and surprising rulings.
- Decision rule
- A rule for choosing between labels, often with a priority order.
Remember: Skim, read definitions and rules closely, try the examples, build a cheat sheet, and check the changelog daily.
Lesson 2: Applying rules precisely
Goal: Apply rule priority, scope and literal definitions, and name the rule behind each label.
Most errors by careful people come from three traps. Skipping the priority order: many guidelines say one rule beats another ("escalation beats every other label"). If an item triggers two rules, the higher one wins, even if the lower one describes it better. Reading in extra meaning: "Ok." is not a happy customer or an annoyed one; it is a message with no stated opinion. Label what is written. Forgetting the scope: a review that hates the weather but loves the curry is positive, if the guideline says only opinions about the order count.
Good annotators can always say *which rule* decided an item. That habit catches mistakes, because if you cannot name the rule you may be guessing, and it makes your notes and escalations useful to reviewers.
Worked example: two rules at once
Guideline: "R3. MIXED: at least one clear positive and one clear negative opinion about the order or service. R7. ESCALATE (do not label) any message with a threat of harm or a person's full phone number. R7 beats every other rule." Message: "Food was fine. The driver was rude and I'm going to find where he lives."
- "Food was fine" is mild praise and "the driver was rude" is a complaint, so R3 alone would give MIXED.
- "I'm going to find where he lives" reads as a threat of harm, which triggers R7.
- R7 beats every other rule.
Label: ESCALATE, because of R7. Put the rule number in your note.
Hands-on lab in the room: Guideline edge-case lab. Ten messages and one guideline. After each answer, the feedback quotes the rule that decides it.
- Rule priority
- An order that says which rule wins when two rules apply to the same item.
- Scope (of a label)
- What a label is about: for example, only opinions about the order count, not opinions about the weather.
- Literal reading
- Labelling what the text says, without adding meaning the writer did not express.
Remember: Check the priority, label what is written, stay in scope, and name the rule.
Lesson 3: Edge cases: never guess
Goal: Handle ambiguous, broken and out-of-scope items with the guideline's tools instead of guessing.
An edge case is an item the main rules do not settle easily. Edge cases are where quality is won or lost, and where gold questions are most often placed. The common kinds:
| Kind | Example | Usual tool |
|---|---|---|
| Genuinely ambiguous | "Fine." as a whole review | The guideline's default rule, or an UNSURE flag if there is one |
| Out of scope | A review in a language the task does not cover | A NOT APPLICABLE or SKIP option |
| Broken data | An image that does not load; silent audio | Report it as broken. Never label a guess |
| Two labels fit | A message that is both a question and a complaint | The guideline's priority rule |
| Not covered at all | A kind of content the guideline never imagined | Check the FAQ and changelog, then escalate |
Guessing is the worst choice. A guessed label looks exactly like a confident one in the dataset, so nobody can find it later. A flag, a skip or a question is visible and can be fixed. If the tool has an UNSURE or SKIP option, use it for what it is for, not to avoid hard work: projects notice both too much use and too little.
Edge cases also deserve notes. A one-line comment such as "Applied the R4 default: no opinion stated" helps a reviewer agree with you quickly, and shows a pattern if many people struggle with the same item.
Worked example: three edge cases
- Image task: the photo is completely black. → Mark it broken or unusable. Do not draw anything.
- Sentiment task (English only): the review is in Portuguese. → NOT APPLICABLE, not NEUTRAL.
- Sentiment task: "Ok." → The guideline's default says messages with no opinion are NEUTRAL. Label NEUTRAL, with the note "no opinion stated (default rule)".
Each one uses a tool the guideline provides. None of them is a guess.
- Edge case
- An item the main rules do not settle easily, such as an ambiguous, broken or out-of-scope item.
- Default rule
- A guideline's instruction for what to do when nothing else decides an item.
- UNSURE or SKIP flag
- A tool option for items you cannot label reliably. Use it for its purpose, not to avoid effort.
Remember: Broken: report it. Out of scope: not applicable. Ambiguous: default rule or flag. Never guess.
Lesson 4: When and how to escalate
Goal: Decide what to escalate and write an escalation a project lead can answer quickly.
Escalating means raising an item or a question with the people who run the project, through the channel they set up: a flag button in the tool, a project forum, a support ticket or a team lead. It is a normal, expected part of the job, not a sign of weakness.
- Escalate: a gap in the guideline or two rules that contradict each other; content the guideline says to escalate, such as threats, personal data or illegal material; broken tasks; anything that might put a real person at risk.
- Do not escalate: items the guideline, the examples or the FAQ already answer (search first); questions about your own pay or account (use the support channel for those); items that are simply hard.
A good escalation lets someone answer in under a minute. Give the task or item ID, quote the rule you tried to apply, say what makes it unclear, and give your proposed label with your reason. Keep it neutral and short. Then keep working: follow the guideline's default for that kind of item, or skip it if the tool allows, until you get an answer.
Item 48213. R3 says MIXED needs "one clear positive and one clear negative opinion about the order or service".
The message: "Driver was nice. Shame about the rain."
Unclear: is the rain a complaint about the service (was the driver late because of it?) or only about the weather?
Proposed: POSITIVE, treating the rain as out of scope under R1. Please confirm.
Watch out: Never discuss task content on public forums, social media or chat groups, even to ask a question. Task data is confidential (Unit 7), and posting it can end your contract. Use only the project's official channels.
Worked example: turning a vague question into a good one
- Vague: "The sarcasm rule is confusing??"
- Better: "Item 9917. R2 says to label sarcasm by intended meaning. The message is 'Wow, only 40 minutes late this time.' It could be a friendly joke or a real complaint."
- Add your proposal: "Proposed: NEGATIVE, because lateness is a service complaint under R2."
An ID, the rule, the unclear point and a proposal. That is all it takes.
- Escalation
- Raising an item or question with the project team through its official channel.
- Guideline gap
- A situation the guideline does not cover, or covers with rules that contradict each other.
Remember: Search first. Then give the item ID, the quoted rule, what is unclear and your proposal, in official channels only.
Practise it in AI Data Annotation & Model Rating
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Guideline edge-case lab.
Every lesson is free: no ads, no lives and no paywall.