Annotation is the process of adding structured labels to raw data, such as images, video, and point clouds, so a machine learning model can learn from it. The labels encode the task: class tags for classification, boxes for detection, or masks for segmentation.
Annotation produces the supervised signal a model trains against. You define a schema or ontology that fixes the classes and attributes, label the raw data against it, review for quality, then export to a standard format like COCO or YOLO for training and evaluation. Each label maps to a model output, a class logit, a box regression, a per-pixel prediction, which is why the annotation format and the model architecture are chosen together rather than in isolation. In visual AI the data is images, video, LiDAR, and other modalities, and in practice the term is used interchangeably with labeling.
Key takeaways
Annotation defines the label space and target format a model learns to predict, so it sets the ceiling on what the model can learn.
Label cost scales with task complexity. A segmentation mask can cost roughly an order of magnitude more than a bounding box, and expert domains like medical imaging cost more still.
A model is only as accurate as its labels, which makes review, inter-annotator agreement, and curation first-class concerns, not afterthoughts.
Types of annotation
Image classification: one or more class tags for the whole image.
Bounding box: a rectangle around an object, stored as coordinates.
Polygon and segmentation mask: object shape at the pixel level, either semantic (a class per pixel) or instance (per-object).
Keypoints: ordered landmarks such as joints, with visibility flags.
Cuboid or 3D box: position, size, and orientation in a point cloud.
Polyline: ordered vertices for continuous features like lane lines.
How it works, and how FiftyOne fits
The workflow is consistent: collect, label, review for quality, then train and evaluate. Modern tools speed the labeling with hotkeys, interpolation, and model-assisted suggestions. FiftyOne wraps the loop around your annotation tool. You curate the slices actually worth labeling, send them to your existing annotation tool, and pull the finished labels back to inspect quality and compare against model predictions before they ever reach training.
The two are used interchangeably in practice. The distinction is emphasis, not a hard boundary.
Why annotation matters
Label errors become label noise, and label noise propagates straight into model behavior, which is why teams measure agreement with metrics like IoU and inter-annotator agreement. There is also an economic limit worth naming: labeling cost grows roughly linearly with dataset size while marginal model gains diminish, so each additional label eventually returns less. Redundant or low-information samples, near-duplicates and easy cases the model already handles, spend budget without moving accuracy. This is why curating informative samples, by embedding distance, model uncertainty, or failure analysis, often beats labeling more data.
Frequently asked questions
What is the difference between annotation and labeling?
Annotation and labeling are largely synonymous. Annotation is the broader term for adding structured information, while labeling often refers specifically to assigning class tags.
What are the main types of annotation?
Classification tags, bounding boxes, polygons and masks, keypoints, cuboids, and polylines.
Why is annotation quality so important?
Annotation quality is important because a model cannot be more accurate than its labels, so annotation errors set a ceiling on performance.
How do you reduce annotation cost?
Curate before you label, prioritizing informative, non-redundant samples over labeling everything.