Skip to content
FORGENO ADS GAMESUnlock all $8

AI Data Annotation & Model Rating · Unit 2 of 8

Unit 2: Annotation types

Classification and multi-label tags, boxes, polygons and segmentation, entity spans, sentiment, transcription and ranking, with the exact rules each one needs.

Opens the first lesson of this unit in AI Data Annotation & Model Rating. Every lesson is open from the start.

Lesson 1: Classification: one label or many

Goal: Choose correctly between single-label and multi-label tasks, and use catch-all labels properly.

Classification means putting each item into a category from a fixed list. In a single-label task the categories exclude each other, so each item gets exactly one, such as POSITIVE, NEGATIVE or NEUTRAL. In a multi-label task an item can have several labels at once, such as a photo tagged both "beach" and "dog", or a support ticket tagged both BILLING and ACCOUNT.

Read the instruction to see which kind you have. Two common mistakes: forcing one label when the task allows several, which throws information away, and ticking several when the task wants the single best one.

TaskKindWhy
Is this email spam?Single-label (two options)Spam and not spam exclude each other
Which topic is this article mainly about?Single-label"Mainly" asks for one
Which topics does this article mention?Multi-labelAn article can mention several
Which problems does the photo show: blur, glare, cropped?Multi-labelA photo can be blurry and cropped

Catch-all labels such as OTHER, NONE or NOT APPLICABLE exist for items that fit no category. Use them only when the categories truly do not fit, not as a shortcut for hard items. Many projects track how often each person uses OTHER, because overuse hides decisions the data needed. Some category lists are hierarchical: first a broad class (Animal), then a finer one (Dog), then finer still (Breed). Get the broad level right before worrying about the fine one.

Worked example: tagging a support ticket

Task: "Tag every issue the customer raises. Labels: BILLING, DELIVERY, PRODUCT_FAULT, ACCOUNT, OTHER." Ticket: "I was charged twice, and the kettle that finally arrived does not switch on."

  1. This is multi-label: "every issue".
  2. Being charged twice is BILLING.
  3. A kettle that does not switch on is PRODUCT_FAULT.
  4. "Finally arrived" hints that delivery was slow, but the customer does not complain about it, so DELIVERY is not tagged.

Labels: BILLING and PRODUCT_FAULT. Tag what is there, not what you imagine.

Classification
Assigning each item to one or more categories from a fixed list.
Single-label task
A task where each item gets exactly one category (also called multi-class when there are more than two).
Multi-label task
A task where an item can have several categories at once.
Catch-all label
A label such as OTHER or NONE for items no category fits. Overusing it hides real decisions.
Taxonomy
The organised list of categories for a task, sometimes in levels from broad to fine.

Remember: "Mainly" means one label; "every" or "all that apply" means several. Use OTHER only when nothing fits.

Lesson 2: Boxes, polygons and segmentation

Goal: Draw tight boxes, apply occlusion and truncation rules, and calculate IoU.

For images and video, the most common annotation is the bounding box: the smallest upright rectangle that contains every visible part of an object. "Tight" is the key word. A box with a margin of road or sky around a car teaches the model that road is part of a car.

Guidelines always say what to do with hard cases, and you must read yours. Typical rules: an occluded object, partly hidden behind something, is boxed around its visible part only, or around its estimated full shape, depending on the project. A truncated object, cut off by the edge of the image, is boxed up to the edge. Pictures of objects (a car on a billboard), reflections and toys are usually not the real class. Very small or very blurry objects may have a minimum size below which you skip them.

Polygons trace an object's outline with many points, for shapes a box fits badly, such as a person with arms out or a river. Segmentation labels every pixel: semantic segmentation gives each pixel a class (road, car, sky), while instance segmentation also separates each object (car 1, car 2). Keypoints mark specific points, such as the joints of a body.

Projects score boxes with IoU, intersection over union: the area where your box and the gold box overlap, divided by the total area the two boxes cover. 1.0 is a perfect match and 0 means no overlap. Projects often count a box as correct above a threshold such as 0.5, and stricter ones use 0.7 or more.

Worked example: IoU by hand

  1. The gold box runs from x = 20 to 60 and from y = 10 to 50: 40 × 40 = 1,600 square units.
  2. Your box runs from x = 30 to 70 and from y = 10 to 50: also 1,600.
  3. They overlap from x = 30 to 60 (30 wide) and y = 10 to 50 (40 tall): 30 × 40 = 1,200.
  4. Union = 1,600 + 1,600 − 1,200 = 2,000.
  5. IoU = 1,200 ÷ 2,000 = 0.6.

Shifting a box by a quarter of its width already drops IoU to 0.6. Tightness matters.

Hands-on lab in the room: Bounding-box labeller. Draw boxes on three scenes and see your IoU against the gold boxes. Each scene has an edge case, so read its rule first.

Bounding box
The smallest upright rectangle that contains every visible part of an object.
Occlusion
When part of an object is hidden behind something else.
Truncation
When an object is cut off by the edge of the image.
Polygon annotation
Tracing an object's outline with many points.
Semantic segmentation
Labelling every pixel with a class, such as road or sky.
Instance segmentation
Labelling every pixel and also telling apart separate objects of the same class.
IoU (intersection over union)
The overlap area of two shapes divided by the total area they cover. 1.0 is a perfect match.

Remember: Tight boxes, your project's occlusion rule, and no boxes on pictures of things. IoU = overlap ÷ union.

Lesson 3: Named entities and text spans

Goal: Tag entity spans with exact boundaries and the right type, and score them with precision and recall.

In text, a lot of annotation means marking spans: a stretch of words with a label. The best-known kind is named-entity recognition (NER): tagging the names of people (PERSON), organisations (ORG), places (LOCATION), dates (DATE) and so on.

Span tasks are judged on two things at once: the type and the exact boundaries. "Dr. Maya Okafor" might be tagged PERSON as "Maya Okafor" (without the title) or as "Dr. Maya Okafor" (with it). Either could be right, but the guideline picks one and everybody must follow it. A span that drops "Inc." from "Brightwater Foods Inc." is a boundary error even though its type is right.

Context decides the type. "Jordan" can be a person or a country; "Washington" a person, a city, a state or a sports team; "Apple" a fruit or a company. Read the whole sentence. Pronouns ("she", "they") are usually not tagged, and generic phrases ("the company", "a doctor") are not names.

Span work is scored with precision (of the spans you tagged, how many were exactly right) and recall (of the gold spans, how many you found). Their combination, the F1 score, is 2 × P × R ÷ (P + R).

Worked example: scoring spans

  1. The gold answer has 4 entities. You tagged 5 spans.
  2. 3 of yours match a gold span exactly: same start, same end, same type.
  3. Precision = 3 ÷ 5 = 0.6. Recall = 3 ÷ 4 = 0.75.
  4. F1 = 2 × 0.6 × 0.75 ÷ (0.6 + 0.75) = 0.9 ÷ 1.35 ≈ 0.67.

Extra spans cost precision; missed spans cost recall. Exact boundaries count for both.

Hands-on lab in the room: Entity span tagger. Tag people, organisations, places and dates in four sentences, with a short guideline. Each sentence is scored against gold spans.

Span
A continuous stretch of text, marked by where it starts and ends, with a label.
Named-entity recognition (NER)
Tagging names of people, organisations, places, dates and similar things in text.
Boundary error
A span with the right type but the wrong start or end, such as one that drops "Inc.".
Precision
Of the items you marked, the share that were right.
Recall
Of the items that should have been marked, the share you found.
F1 score
A single score combining both: 2 × precision × recall ÷ (precision + recall).

Remember: Right type, exact boundaries, decided by context. Extra spans cost precision; misses cost recall.

Lesson 4: Sentiment, transcription and ranking

Goal: Apply sentiment scales, transcription conventions and ranking methods.

Sentiment tasks label the opinion a text expresses. Scales vary: three labels (positive, negative, neutral), often with MIXED added, or five points from very negative to very positive. Aspect-based sentiment labels the opinion about each thing separately: "Great screen, terrible battery" is positive for the screen and negative for the battery. Label the opinion the writer expresses, not whether you agree with it, and judge sarcasm by what the writer means.

Transcription turns speech into text. The guideline picks a style. Verbatim keeps everything, including "um", repeats and false starts; clean verbatim removes fillers but keeps the meaning. It also fixes conventions: tags such as [inaudible], [crosstalk] or [laughter], how to write numbers, and how to mark speakers. Never guess a word you could not hear: tag it.

AudioVerbatimClean verbatim
"So, um, the the meeting is at, uh, three."So, um, the the meeting is at, uh, three.So the meeting is at three.
A word drowned out by a siren[inaudible][inaudible]

Ranking tasks put several items in order: search results by relevance, or four chatbot answers from best to worst. A ranking is really a set of pairwise judgments (A beats B, B beats C). Good rankers fix the clearly best and clearly worst first, then compare the close middle ones directly, and use ties only if the tool allows them.

Worked example: aspect-based sentiment

Review: "The hotel room was spotless and the bed was comfy, but breakfast was cold and overpriced." Aspects: ROOM, BREAKFAST, STAFF.

  1. ROOM: "spotless" and "comfy" → POSITIVE.
  2. BREAKFAST: "cold" and "overpriced" → NEGATIVE.
  3. STAFF: not mentioned → NOT MENTIONED (or blank), as the guideline says. Do not guess.

The review as a whole might be MIXED, but the aspect labels keep the useful detail.

Sentiment analysis
Labelling the opinion a text expresses, such as positive, negative, neutral or mixed.
Aspect-based sentiment
Labelling the opinion about each feature or topic separately.
Verbatim transcription
Writing down speech exactly, including fillers, repeats and false starts.
Clean verbatim
Transcription that removes fillers and false starts but keeps the meaning.
Ranking
Putting several items in order of quality or relevance.

Remember: Label the opinion expressed, tag what you cannot hear, and rank the ends first.

Practise it in AI Data Annotation & Model Rating

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 7-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Bounding-box labeller and Entity span tagger.

Every lesson is free: no ads, no lives and no paywall.