Inter-annotator agreement (IAA) is a measure of how consistently different annotators assign the same labels to the same data. High agreement signals clear guidelines and reliable ground truth, while low agreement reveals ambiguity in the task, the schema, or the instructions.
Inter-annotator agreement, also called inter-rater agreement or annotator consensus, quantifies how much independent annotators agree when they label the same items. You have several people label an overlapping set, then compute how often they match. It is the primary way to gauge whether your labels are reproducible: if two qualified annotators routinely disagree, the "right answer" is not well defined and the resulting ground truth is shaky.
Agreement is reported with metrics that correct for chance, so random matching does not inflate the score.
Key takeaways
IAA measures label consistency across annotators, a proxy for how reliable the ground truth is.
Chance-corrected metrics like Cohen's kappa and Fleiss' kappa keep agreement from being overstated.
Low agreement usually points to ambiguous guidelines or class boundaries, not careless annotators.
What inter-annotator agreement provides
Common agreement metrics and what each one is for.
Common agreement metrics and what each one is for.
Metric
What it measures
Cohen's kappa
Agreement between two annotators, corrected for chance
Fleiss' kappa
Agreement among three or more annotators
Krippendorff's alpha
Agreement across data types and with missing data
Raw percent agreement
Simple overlap, but ignores chance
How it works
You assign overlapping samples to multiple annotators and compare their labels, then compute a chance-corrected score. In FiftyOne you can store multiple label sets on the same samples and surface where they disagree, turning agreement from a single number into a list of the specific items worth re-examining. For the formal definitions and when to use each metric, see Artstein and Poesio (2008).
Why it matters
IAA is the cheapest early-warning system for label quality, so you can catch a broken guideline before you have paid to mislabel an entire dataset. It is also a ceiling on achievable model accuracy. If humans only agree 80% of the time, a model that "exceeds" that is often just fitting one annotator's idiosyncrasies, so reported accuracy above the human agreement rate should be treated with suspicion rather than celebrated. Disagreement is signal, because the samples annotators split on are usually the same ones models find hardest.
Frequently asked questions
What is a good inter-annotator agreement score?
It depends on the metric and task, but kappa above roughly 0.8 is often considered strong, and below about 0.6 signals real ambiguity.
Why correct for chance?
Because annotators can match by luck, especially with few classes, and chance-corrected metrics like kappa remove that.
What does low agreement tell you?
Usually that the guidelines or class definitions are ambiguous. Fix the instructions before blaming annotators.