Key takeaways
- Data labeling is the process of attaching ground-truth annotations to raw data so supervised machine-learning models can learn from it.
- Quality and consistency of labels often matters more to model performance than model architecture or hyperparameter tuning.
- Annotation methods vary widely: classification labels, bounding boxes, segmentation masks, ranked preferences, transcripts, entity tags, ratings.
- Crowdsourcing (Amazon Mechanical Turk, Scale AI, Appen) scales volume; domain experts handle specialized tasks.
- Programmatic labeling, active learning, and LLM-assisted annotation are increasingly important for reducing the human-hours needed.
Why labels matter more than models
When a model underperforms, the instinct is often to try a bigger architecture or more hyperparameter tuning. In practice, bad labels cap performance in a way no model change can overcome. If 10% of your training set has wrong labels, no model can exceed roughly 90% accuracy — the ceiling is set by the data.
Andrew Ng’s 2021 talk “A Chat with Andrew on MLOps” popularized the framing of data-centric AI: for most production problems, improving the data is a better investment than improving the model. The underlying intuition is backed by studies showing that small fixes to labeling consistency frequently beat major architectural changes on benchmark tasks. See our machine learning primer for the underlying theory.
Common annotation types
Classification labels
The simplest — for each example, pick one or more categories. Spam or not. Topic from a list of 50. Sentiment positive / neutral / negative. Fast to produce, limited in expressive power.
Bounding boxes
For object detection in images and video, draw a rectangle around each object and label it. Used to train detection models for self-driving perception, retail inventory counting, security surveillance. One image often has 10-50+ boxes and takes 30-120 seconds per image.
Segmentation masks
Pixel-level labeling — each pixel gets a class. Semantic segmentation labels by category; instance segmentation separates individual objects. Much slower than bounding boxes — minutes per image — but necessary for applications needing precise object boundaries (medical imaging, satellite analysis).
Keypoints and pose
Mark specific points on an object (joints of a human body, landmarks on a face). Used for pose estimation, action recognition, animation. See our computer vision coverage for the computer-vision context.
Text annotation
Entity recognition (tag mentions of people, places, dates). Intent classification (which of N intents does this utterance express). Coreference (which pronouns refer to which entities). Summarization or paraphrasing (write a label text). Text annotation ranges from fast (per-sentence labels) to slow (multi-paragraph translation).
Preference ranking
Show two or more model outputs, rank them. Critical for RLHF (see our RLHF primer) and increasingly for LLM post-training in general. Easier than writing new content but requires careful guidelines to get consistent rankings.
Transcripts and alignment
For speech and audio, type what was said. For alignment tasks, mark timestamps where specific events occur. Podcast transcription, medical dictation, and meeting summaries rely on large transcription datasets.
Who does the labeling
Crowdsourcing platforms
Amazon Mechanical Turk pioneered the large-scale model. Modern platforms like Scale AI, Appen, Labelbox, and SuperAnnotate manage crowd workforces plus tooling. Costs range from a few cents per simple label to dollars for complex segmentation. Quality varies widely — redundancy (multiple labelers per example) and quality checks are essential.
In-house annotation teams
For proprietary or sensitive data, many companies build in-house teams. Higher cost per label but better quality control, better handling of confidential data, and steeper learning curves on domain-specific tasks. Major AI labs run large internal annotation operations.
Domain experts
Medical imaging requires radiologists. Legal document classification needs lawyers. Scientific literature classification needs PhDs. Expert labels are 10-100x more expensive than crowd labels but are often the only path to usable models for specialized tasks.
Model-assisted labeling
A growing category. A model makes initial predictions; humans review and correct. Throughput can be 5-10x higher than from-scratch labeling. Works best for mature tasks where the model is already reasonably accurate. Active learning — training on labeled examples, using the model to pick the most informative unlabeled examples for human review, iterating — has become standard in modern labeling pipelines.
Quality control
Labeling guidelines
The single biggest quality lever. A good guideline document is 20-200 pages with unambiguous definitions, dozens of edge cases, and explicit examples of correct and incorrect labels. Ambiguity in guidelines produces ambiguity in labels.
Inter-annotator agreement
Have multiple annotators label the same data. Measure agreement with Cohen’s kappa or Krippendorff’s alpha. Low agreement signals unclear guidelines or inherently subjective tasks. High agreement is necessary but not sufficient — annotators can agree on the wrong label if the guidelines are flawed.
Gold-standard examples
A held-out set of expert-verified labels used to test annotators. Annotators who fail the gold set are retrained or removed. Platforms like Scale and Appen build this into their workflows.
Consensus aggregation
For each example, combine multiple labels. Majority vote, weighted voting based on annotator reliability, or probabilistic models (Dawid-Skene, MACE) that estimate annotator accuracy. See our model training guide for how labels feed into training.
Auditing samples
Ongoing random sampling of completed labels, reviewed by senior annotators or engineers. Catches drift in labeler behaviour and flags guideline gaps.
Costs
Labeling is often the largest single cost in an AI project. A labeled image classification dataset of 100,000 images might cost $5,000-$20,000. A labeled self-driving perception dataset can cost tens of millions of dollars — large autonomous-vehicle programs have spent hundreds of millions cumulatively. For RLHF post-training, preference-labeling budgets at major labs run into the tens of millions.
Cost drivers include complexity per example (classification vs. segmentation), expertise required, quality-control overhead, and project management. Plan for labeling costs to exceed model training costs on most non-trivial projects.
Emerging approaches
Synthetic data
For some domains — self-driving simulations, augmented images for rare categories, programmatically-generated question-answer pairs — synthetic data can substitute for real labeled data. Quality depends heavily on the domain. Purely synthetic training rarely matches real data; hybrid approaches are common.
LLM-as-labeler
Large language models can generate labels for many text tasks at per-item costs below crowd workers. Quality is comparable to non-expert humans for common tasks, lower for specialized ones. Widely used for initial drafts that humans verify, and for building preference datasets at scale. Major labs have documented using LLMs to label training data for smaller specialized models.
Weak supervision
Tools like Snorkel let domain experts write labeling heuristics — programmatic rules that assign probabilistic labels to many examples at once. A human-written rule might be imperfect; multiple rules combined statistically can approximate high-quality labels at scale. Useful when high-quality gold labels are expensive but cheap rules are easy to write.
Ethical considerations
The annotation industry has faced scrutiny over worker pay, working conditions, and exposure to disturbing content (content moderation datasets). Labelers processing graphic violence, child sexual abuse material, and hate speech have experienced documented mental-health impacts. Responsible labeling programs include clear pay standards, psychological support for content-moderation work, and diverse labeler pools to reduce bias in labels.
Frequently asked questions
Can I skip labeling and just use an LLM?
For many text tasks, yes — at a quality cost. LLMs can classify, extract, and rate text without task-specific fine-tuning, often matching crowd-worker quality. For specialized or high-stakes tasks, LLM labels still need expert verification. The realistic pattern is LLMs generating draft labels that humans review, compressing the work without eliminating it.
How many labeled examples do I need?
Depends on the task and model. Fine-tuning a pre-trained language model on a simple classification task can work with a few hundred examples. Training a specialized object detector from scratch needs tens of thousands. Rule of thumb: start with a few hundred, evaluate, and add more in the regions where the model underperforms. Active learning helps allocate additional labeling effort where it matters most.
What is the biggest labeling mistake teams make?
Underestimating guidelines. Teams assume “it’s obvious what the label should be” and produce a two-page guideline. Labelers interpret edge cases differently, inter-annotator agreement is low, and the trained model is unpredictable. Investing more time in guidelines, piloting with a small batch, iterating on the guideline before scaling, and auditing ongoing work pays off disproportionately.
