AI Data Annotation & Model Rating · Unit 7 of 8
Unit 7: Safety, wellbeing and confidentiality
Labelling against a safety policy, handling disturbing material while protecting your wellbeing, keeping task data confidential, and balancing speed with quality.
Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.
Lesson 1: Safety labelling
Goal: Label content against a safety policy by judging context and severity, and rate model responses for both unsafe help and over-refusal.
Safety projects label content, both what people write and what models produce, against a content policy: a written list of harm categories with definitions, examples and severity levels. Typical categories include harassment and bullying, hate (attacking people for who they are), violence and threats, self-harm, sexual content, dangerous activities such as weapons or drugs, and fraud or scams. Every project's policy is different, so you learn it like any other guideline.
Context decides. The same words can be harmful or harmless, and most policies draw the same distinctions:
- Endorsing or encouraging harm ("You should hurt them") versus describing or reporting it ("The article reports that two people were hurt").
- Instructions that make harm easier versus safety information ("Never mix bleach and ammonia: together they release a toxic gas").
- Attacking people for who they are versus discussing a group, an idea or a policy.
- Fiction, education and history versus real calls to action, though many policies still limit graphic detail.
Policies also grade severity, for example none, low, medium and high, and treat imminent risk, anything suggesting a person is in danger right now such as a credible threat or someone at risk of self-harm, as the top priority with its own escalation route. When you rate model responses, apply both halves of the standard: a response that gives real help with a harmful request is unsafe, and a response that refuses a safe request, such as how to store kitchen knives safely, is over-cautious.
Worked example: labelling against a policy
Policy excerpt: "VIOLENCE: content that threatens, incites or glorifies violence against a real person or group. Reporting, history and clearly fictional violence are not violations unless they include instructions for causing harm. Severity HIGH if a specific, credible threat is made."
- "A huge earthquake and fire destroyed much of the city in 1906." → No violation: history, and not violence by people.
- "In my novel, the knight defeats the dragon in a fierce battle." → No violation: clearly fiction, with no instructions.
- "I know where you park. You'll regret Friday." → VIOLENCE, severity HIGH: a specific, credible-sounding threat to a real person. Escalate it as the policy requires.
Words like "destroyed", "battle" and "regret" decide nothing on their own. Who is targeted, whether it is real, and whether it threatens or instructs do.
- Content policy
- A written set of harm categories, definitions, examples and severity levels used for safety labelling.
- Severity level
- How serious a policy violation is, for example low, medium or high.
- Endorsing versus describing
- The difference between encouraging harm and reporting or discussing it. Most policies treat them very differently.
- Imminent risk
- Content suggesting someone may be in danger right now. Policies make it the top priority, with its own escalation route.
Remember: Learn the policy, then judge context: endorse or describe, instruct or inform, real or fiction. Grade the severity, and escalate imminent risk.
Lesson 2: Disturbing content and your wellbeing
Goal: Know what responsible projects do before and during disturbing work, and protect your own wellbeing.
Some projects involve disturbing content: violence, abuse, hate, self-harm or graphic images. The work protects people, because it trains systems that catch such content, but it can affect the people who do it. Repeated exposure can bring stress, poor sleep, intrusive images or numbness, sometimes called secondary (vicarious) trauma. Knowing this in advance is part of doing the job well and safely.
What responsible projects do. Practices vary, but the good ones share these habits:
- They tell you before you start what kinds of content may appear, so you can give informed consent.
- They let you opt out of a project or a content category, or skip a task, without penalty.
- They offer tools such as blurring, greyscale or muted audio for graphic material.
- They limit exposure time and encourage breaks.
- They provide wellbeing support, such as counselling, or tell you where to find it.
Exact policies differ by company, contract and country, so read your own project's terms. A project that springs graphic content on you without warning, or punishes you for skipping it, is a warning sign. Your own habits matter too: take regular breaks away from the screen; keep this work away from bedtime if you can; notice changes in your sleep or mood; talk to someone you trust, without sharing confidential task details; and stop or switch projects if it is hurting you. No task is worth your health, and a strained rater makes worse judgments.
Watch out: If task content ever leaves you feeling unsafe or thinking about harming yourself, stop and reach out. In the US you can call or text 988 (the Suicide & Crisis Lifeline) at any time. Elsewhere, contact your local emergency number or a crisis line.
Worked example: a healthy routine on a safety project
- Before accepting: read the content warning and the opt-out rules, and ask yourself honestly whether this is right for you now.
- While working: use the blur tool, work in blocks with real breaks between them, and stop at a set time each day.
- Afterwards: switch to something unrelated and physical, such as a walk, cooking or music.
- Each week: check in with yourself. Sleeping badly or feeling numb for more than a few days means switching projects and using the support on offer.
Protecting your wellbeing is not weakness. It keeps your judgment sharp, which is what the project needs from you.
- Disturbing content
- Material such as violence, abuse, hate or self-harm that can upset the people who review it.
- Secondary (vicarious) trauma
- Stress reactions that come from repeated exposure to other people's traumatic material.
- Informed consent
- Agreeing to something after being told clearly what it involves, including its risks.
- Opt-out
- The ability to decline a project, a content category or a task without penalty.
Remember: Know what you are agreeing to, use opt-outs and blur tools, take breaks, and stop if the work is hurting you.
Lesson 3: Privacy, confidentiality and NDAs
Goal: Keep task data confidential, recognise personal data, and know what you may and may not say about your work.
Almost every AI data job comes with a confidentiality agreement, often called a non-disclosure agreement (NDA). It usually covers the tasks, the guidelines, the client's name and anything you learn about their products. Breaking it can end your contract at once and can have legal consequences. The habits that keep you safe are simple, but they must become automatic.
- Never paste task data anywhere else: not into a search engine, a translation site, another chatbot or a writing tool, unless the guideline explicitly allows that specific tool. Whatever you paste may be stored by that service.
- No screenshots or copies of tasks or guidelines, unless the project asks for them in a report.
- No public talk: do not name the client, describe tasks or share guidelines on social media, forums, videos or group chats. "Everyone already knows" is not permission.
- Work privately: on your own device and account, out of sight of others, and never share your login.
- Handle personal data with care: if a task contains someone's real name, address, health details or ID numbers, do only what the task requires, never copy or keep it, and use the project's flag for personal data if it has one.
Personally identifiable information (PII) is anything that can identify a person: names, phone numbers, email and home addresses, ID numbers, faces and voices, and combinations such as "the only dentist in a small town". Some projects pay you to find and redact it; others ask you to flag it. When you write prompts or ideal responses, invent details ("Sam from Riverton") instead of using real people.
Worked example: three moments of temptation
- A long legal passage in a task is confusing. Pasting it into a free chatbot to summarise it would leak client data. Instead, use the guideline's help section or escalate a question.
- A friend asks what you do. If your agreement allows that much, you can say "I rate AI responses as a contractor", but not which client or what the tasks contain.
- A transcription task includes a caller reading out a full card number. You follow the guideline, often by writing a tag such as [REDACTED], and flag the item. You never write the number down anywhere else.
When in doubt, keep it inside the project's tool and ask through official channels.
- Non-disclosure agreement (NDA)
- A contract in which you promise to keep certain information confidential.
- Confidential information
- Information you must not share outside the project, such as tasks, guidelines and the client's identity.
- Personally identifiable information (PII)
- Any information that can identify a person, alone or combined with other details.
- Redaction
- Removing or masking personal or confidential details, for example replacing a phone number with [PHONE].
Remember: Task data never leaves the task tool: no pasting into other tools, no screenshots, no public talk. Personal data is flagged or redacted, never copied.
Lesson 4: Speed versus quality
Goal: Read time targets and quality scores together, raise speed without losing accuracy, and report time honestly.
Most projects track both how good your work is and how long it takes. You may see a target time per task or an average handling time (AHT), and some projects cap the time you can record for each task. Speed matters, because clients pay for finished work. Quality decides whether you keep getting tasks: falling below the gold-accuracy or audit threshold can mean losing access to a project, however fast you are.
| Pattern | What usually happens |
|---|---|
| Fast and careless | Gold failures and audit errors; access reduced or removed |
| Slow and careful | Good scores, but missed time targets, and time far above the expected range can be questioned |
| Careful first, then faster | The sustainable path: accuracy stays high while your time falls with practice |
| Gaming the timer | Clicking through tasks, or inflating the time you record, shows up in audits and can end a contract |
Quality first, then speed. In your first days on a project, work slowly, reread the guideline and keep your cheat sheet beside you. Speed comes from familiarity, not rushing: as you learn the common item types you recognise them faster. Track yourself honestly by noting your time and your quality feedback for each session, and speed up only while quality holds. Record your time truthfully. Recording more time than you worked is billing for work you did not do; recording less to look fast hides real problems, such as a task type that takes longer than the project expected.
Worked example: reading your own numbers
- Week 1: 30 tasks in 3 hours, so 6 minutes each, with gold accuracy of 96%.
- Week 2: you push to 4 minutes each, 45 tasks in 3 hours, and gold accuracy drops to 85%, below the project's 90% line.
- You ease back to 5 minutes each: 36 tasks in 3 hours, and accuracy returns to 95%.
- Over the following weeks your time falls naturally towards 4 minutes while accuracy stays above 94%.
Speed you have to force costs quality. Speed that comes from practice does not.
- Average handling time (AHT)
- The average time spent on each task.
- Throughput
- How many tasks you complete in an hour or a session.
- Time cap
- The most time a project allows you to record for one task.
- Accurate time reporting
- Recording exactly the time you actually worked, neither more nor less.
Remember: Quality first. Let speed come from practice, track both honestly, and never game the timer.
Practise it in AI Data Annotation & Model Rating
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit.
Every lesson is free: no ads, no lives and no paywall.