Annotation QA, or quality assurance, is the review process that checks and corrects labels before they reach training, catching errors, enforcing consistency, and verifying completeness. It is how annotation quality is actually achieved in practice, the operational counterpart to measuring quality.
Annotation QA, quality assurance, and review is the step where finished labels are inspected and fixed rather than trusted blindly. Where annotation quality is the property you want and metrics like inter-annotator agreement measure it, QA is the workflow that produces it: a reviewer, a model, or both, checks labels for wrong classes, sloppy boundaries, inconsistencies, and missing objects, then sends problems back for correction.
It can be full review (every label checked), sampled review (a representative subset), or targeted review (only the labels most likely to be wrong), and it usually runs continuously, not just at the end.
Key takeaways
QA is the process of reviewing and correcting labels, the operational side of annotation quality.
It can be full, sampled, or targeted, depending on cost and risk.
Targeted review, focusing on likely-wrong labels, gets the most quality per hour spent.
What annotation QA provides
Common annotation review strategies.
Common annotation review strategies.
Strategy
What it is
Full review
Every label checked, highest cost, used for high-stakes data
Sampled review
A representative subset, estimates quality and catches systemic issues
Targeted, model-assisted review
Surface the labels a model flags as likely wrong, review those first
Consensus and adjudication
Multiple annotators plus a tiebreaker for ambiguous cases
How it works
Reviewers work through a queue of labels, accept or fix each, and route disputes for adjudication, ideally with the riskiest labels surfaced first. FiftyOne supports this directly: rank samples by model mistakenness or disagreement, compare labels to predictions, and pull the suspicious ones to the top so review effort lands where it matters.
Why it matters
QA is where annotation quality is won or lost, because labels are never right by default at scale. Random review barely moves quality. If 3% of labels are wrong, spot-checking random samples mostly confirms the 97% that are fine. The leverage is in targeting, because model-flagged disagreements, low-agreement items, and high-loss samples concentrate the errors, so the same review hours catch far more mistakes when directed there. A QA program's value is set by how well it routes attention, not by how many labels it re-checks.
Frequently asked questions
What is annotation QA?
The process of reviewing and correcting labels to ensure they are accurate, consistent, and complete before training.
What is the difference between annotation QA and annotation quality?
Quality is the property you want. QA is the review process that achieves it.
How do you make QA efficient?
Target review at the labels most likely to be wrong, using model disagreement, low agreement, and high loss.