AI Data Annotation & Model Rating · Unit 6 of 8
Unit 6: Writing, fact-checking and code review
Writing ideal responses, editing model answers and writing test prompts, fact-checking claims against sources, and the basics of judging a coding response.
Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.
Lesson 1: What an ideal response looks like
Goal: Write responses that answer first, meet every constraint, stay accurate and honest, and decline well when they should.
Many AI trainer tasks ask you to write the response you wish the model had given, or to write a prompt and its ideal response together. These ideal (sometimes called "golden") responses teach the model by example, in a step often called supervised fine-tuning. So they must be better than what the model can already do, not merely acceptable.
| Quality | What it means in practice |
|---|---|
| Answers first | The direct answer comes in the first sentence or two, before any background |
| Meets every constraint | Length, format, audience, language, and anything the prompt said to include or avoid |
| Accurate and checked | Every fact, number, step and line of code verified, not assumed |
| Complete, not padded | Covers what the person needs; no restating the question, no generic filler |
| Clear structure | Short paragraphs, numbered steps for procedures, tables only when they help |
| Honest about limits | Says when something is uncertain, depends on the situation, or needs a professional |
| Right tone | Calm and kind for a worried person, brief for a quick question, never preachy |
Some prompts should not get a full answer: a request that would help someone cause harm, or one the model cannot know, such as the person's own bank balance. An ideal response then declines briefly and helpfully: it says what it cannot do, without lecturing, and offers the closest safe help. For a prompt with a false premise, it corrects the premise kindly, then helps. Your project's style guide settles the details, such as greetings, emoji, markdown and whether the assistant says "I".
Worked example: improving a weak response
Prompt: "My spreadsheet has regions in column A and sales in column B. How do I total the sales for North only?" Weak response: "Great question! Spreadsheets are powerful tools used by millions of people. There are many ways to add up numbers, such as SUM, SUMIF and pivot tables, and the best one depends on your needs."
- Problems: a filler opening, no direct answer, and it never tells the person what to type.
- Ideal first line: "Use SUMIF:
=SUMIF(A:A, "North", B:B)." - Then one sentence on what it does: it adds the values in column B on every row where column A says North.
- Then the one caution most likely to matter: the region text must match, so a cell that says "North " with a trailing space will not be counted.
Three short sentences that answer, explain and prevent the most likely mistake. That is the standard to write to.
- Ideal (golden) response
- A response written to be the best possible answer to a prompt, used to teach a model by example.
- Supervised fine-tuning (SFT)
- Training a model on example prompts paired with high-quality responses written or approved by people.
- Style guide
- A project's rules for how responses should look and sound, such as formatting, tone and greetings.
- False premise
- An incorrect assumption built into a question, such as "Why can you see the Great Wall of China from space?"
Remember: Answer first, meet every constraint, check every fact, cut the filler, and be honest about limits.
Lesson 2: Editing responses and writing prompts
Goal: Make minimal, complete edits to model responses, and write realistic prompts that test a skill.
Rewrite tasks give you a model response and ask you to turn it into an ideal one. The skill is knowing how much to change. Keep what is correct and well put; fix every error; remove what does not belong. A full rewrite of a nearly good response wastes time and throws away good material, while a light polish of a wrong one leaves the error in the data. Many projects also ask you to summarise what you changed.
| What you find | The edit |
|---|---|
| A false fact, number or step | Correct it, and check the fix against a reliable source |
| A missed constraint | Restructure to meet it: cut to the word limit, switch to a table, change the language |
| Filler and repetition | Delete it |
| A confusing order | Reorder: answer, then explanation, then extras |
| The wrong tone for the person | Rewrite those sentences and keep the content |
| Unsafe or overconfident advice | Remove it, or add the right caution or referral |
Prompt-writing tasks ask you to write prompts that test the model. Good prompts are realistic (something a real person would ask), specific, and aimed at a skill the project wants to test, such as multi-step reasoning, juggling several constraints or a tricky edge case. They also have a clear way to tell a good answer from a bad one. Never include real people's private information, and follow the project's rules on topics. A prompt the model already answers perfectly teaches it little, so projects often want adversarial prompts that make the model fail, so that your ideal response can then show it the right way.
Worked example: a minimal, complete edit
Prompt: "List three tips for a first job interview, one line each." Model: "1. Research the company before you go. 2. Arrive 10 to 15 minutes early. 3. Ask about salary in the first five minutes to show confidence. Good luck, you will do great, interviews can be scary but preparation helps a lot!"
- Tips 1 and 2 are sound and one line each: keep them.
- Tip 3 is poor advice for most first interviews. Replace it: "Prepare two or three questions to ask the interviewer."
- The closing line breaks the "one line each" format and adds nothing. Delete it.
- Change note: "Replaced tip 3 (poor advice); removed the closing filler to meet the format."
Two lines kept, one fixed and one removed. The response is now correct and on-format, and it still reads naturally.
- Rewrite task
- A task where you turn a model's response into an ideal one.
- Minimal edit
- Changing only what is wrong or missing, and keeping what is already good.
- Prompt-writing task
- A task where you write realistic prompts that test a particular skill of the model.
- Adversarial prompt
- A prompt written to expose a model's weaknesses, such as a tricky edge case. It is test material, not an attack on real systems.
Remember: Keep what is right, fix what is wrong, cut what does not belong, and note what you changed.
Lesson 3: Fact-checking with sources
Goal: Split a response into claims, check the specific ones against good sources, open every citation, and label each claim.
Accuracy is the dimension raters most often get wrong, because false statements can sound exactly like true ones. Fact-checking is a method, not a feeling:
- Split the response into claims, one checkable statement each: a date, a number, a name, a cause, a quote.
- Decide what needs checking. Basic common knowledge does not need a search; specific figures, dates, quotes, laws, and health or money facts always do.
- Find a good source. Prefer primary and official sources (the organisation itself, a government agency, the original document or study), then reputable reference works and established news outlets. Check dates: when facts change, a newer official source beats an older one.
- Read laterally. To judge an unfamiliar website, leave it and see what other trusted sources say about it, instead of trusting its own "About" page.
- Label each claim: supported, contradicted, or not verifiable with the sources allowed.
Model citations need special care. A model can produce a reference that looks perfect (authors, title, journal, year) but does not exist, or cite a real source that does not say what the model claims. Open every cited source and find the sentence that supports the claim. If you cannot, the citation fails, even if the claim happens to be true. Follow your project's rules on research, too: some projects let you search the web, some give you a fixed set of documents, and some forbid outside tools entirely. Never paste task text into a search engine or another AI tool unless the guideline allows it (Unit 7).
Worked example: checking three claims
Response: "The Golden Gate Bridge opened in 1937, is painted a colour called International Orange, and is the longest suspension bridge in the world."
- "Opened in 1937": the bridge operator's own history pages give May 1937. Supported.
- "International Orange": confirmed by the same official source. Supported.
- "The longest suspension bridge in the world": it held the record for its main span when it opened, but not for decades; reference sources list many longer spans today. Contradicted.
Two of three claims hold, but one false claim is enough to fail accuracy. The note: "Says it is the world's longest suspension bridge; it was when it opened in 1937, but is not now."
Hands-on lab in the room: Fact-check desk. Check six claims from a model's answer against four sources. Choose a verdict for each claim and the source that settles it.
- Claim
- One checkable statement, such as a date, number, name, cause or quote.
- Primary source
- The original or official source of a fact, such as the organisation itself, a government agency or the original study.
- Lateral reading
- Judging an unfamiliar source by checking what other trusted sources say about it.
- Fabricated citation
- A reference produced by a model that does not exist, or does not say what is claimed.
- Unverifiable claim
- A claim that the allowed sources neither support nor contradict.
Remember: Split into claims, check the specific ones against primary sources, open every citation, and label each claim.
Lesson 4: Evaluating code responses
Goal: Check a short coding response for instructions, correctness on edge cases, security and an honest explanation.
Coding prompts are one of the largest categories of AI training work, and many projects want raters who can read code. Even at entry level you should be able to judge a short response in a language you know. The four dimensions still apply; they just become more specific.
| What to check | In a coding response |
|---|---|
| Instruction-following | The right language, version and libraries; exactly the function or behaviour asked for; any constraints, such as no outside libraries |
| Correctness | The right result for normal inputs and for edge cases: an empty input, zero, negative numbers, one item, very large values |
| Safety and security | No passwords or keys written into the code; user input never glued straight into database queries or commands; no destructive commands without warning |
| Honest explanation | The prose describes what the code really does, including its limits |
| Readability | Clear names, sensible structure, comments where they help |
Trace before you trust. Pick two or three small inputs, including an edge case, and work through the code by hand, or run it if your project provides a sandbox and allows it. Do not run model-written code on your own computer when the project forbids it, and never paste task code into public tools. A response whose explanation says "this handles empty lists" when the code crashes on an empty list fails twice: on correctness and on honesty.
Worked example: tracing a function
Prompt: "Write a Python function that returns the average of a list of numbers, and returns 0 for an empty list." The response gives the code below and says: "This safely handles empty lists."
- Normal input
[2, 4, 6]: total is 12, the length is 3, so it returns 4.0. Correct. - Edge case
[]: the loop does nothing, then0 / 0raises a ZeroDivisionError. It crashes instead of returning 0. - The explanation claims empty lists are handled. That is false.
- The fix is one line at the start:
if not nums: return 0.
def average(nums):
total = 0
for n in nums:
total += n
return total / len(nums)Right for normal input, wrong on the very edge case the prompt named, and dishonest about it. The note: "Crashes on [] with ZeroDivisionError, though the prompt requires 0 and the response claims it is handled."
- Tracing (code)
- Working through code by hand, line by line, with a sample input to see what it really does.
- Sandbox
- A sealed environment for running code safely, without access to your own files or systems.
- Hard-coded secret
- A password, key or token written directly into code, where anyone who sees the code can read it.
- Injection
- A security flaw where user input is treated as part of a command or database query, letting an attacker change what it does.
Remember: Check the language and constraints, trace a normal and an edge input, look for secrets and unsafe input handling, and compare the explanation with the code.
Practise it in AI Data Annotation & Model Rating
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Fact-check desk.
Every lesson is free: no ads, no lives and no paywall.