What is the difference between data annotation and data labeling?

In practice, data annotation and data labeling are used interchangeably. When a distinction is drawn, data labeling means assigning tags or classes, and data annotation covers the broader work of adding structured information, including spatial labels such as boxes and masks and temporal labels such as event boundaries.

What is ground truth in machine learning?

Ground truth is the set of labels accepted as correct for a dataset, the reference a model is trained on and evaluated against. Annotation creates ground truth, and annotation QA keeps it trustworthy. Errors in ground truth carry directly into models trained on it.

What is the difference between auto-labeling and agentic labeling?

Agentic labeling is a form of auto-labeling. Standard auto-labeling, also called automated labeling, runs a model over a dataset to generate labels in bulk for people to review. Agentic labeling starts from a natural-language description of the task, which people refine against a few samples before it is applied to the whole dataset and saved for reuse.

What is a label schema?

A label schema is the defined structure of an annotation task: which classes exist, which label types each one uses, and which attributes annotators can set. For example, a schema might say that cars use 3D cuboids with a color attribute and lane markings use polylines. The schema and written guidelines are what keep labels consistent across annotators and over time.

Do teams need to annotate all of their data?

No, teams don't need to annotate all of their data, and trying to is the most common way annotation budgets are wasted. Most raw data is routine, and labeling it adds little. Teams use curation and active learning to select the samples most likely to improve the model, then concentrate annotation effort there.

How are robot episodes annotated?

Robot episodes are annotated mostly with language and time: a task description for the whole episode, subtask events such as reach, grasp, and place, and a success or failure flag. Object tracks and keypoints are added when a model needs them.

Data Annotation for Visual and Physical AI

Q: How does annotation QA work?

The annotation QA process reviews labels before they reach training data. Modern QA ranks the labels most likely to be wrong using embeddings, model predictions, and comparisons against auto-labels, so reviewers work highest-risk first, and teams track inter-annotator agreement against an expert-verified gold set.

Data annotation turns raw images, video, point clouds, and robot recordings into the labeled examples models learn from. This guide covers how annotation works for visual and physical AI: the label types, the workflow from schema design to quality assurance (QA), how robot episodes and multi-sensor recordings are labeled, and how auto-labeling and agentic labeling are changing what human annotators do.

What is data annotation?

Data annotation, also called data labeling, is the process of adding structured labels to raw data so a machine learning model can learn from it. The labels become the ground truth that a model is trained on and measured against, which means label quality sets a hard ceiling on model quality. A model cannot learn distinctions its training labels never made.

This guide covers annotation for visual and physical AI, not language models: marking what exists in a scene, where it is, when it happens, and, for robots, what task was being attempted.

Annotation is one of the most expensive and time-consuming steps in most visual AI pipelines, and much of that spend is wasted. In the 2026 State of Visual and Physical AI survey, 99% of teams described their annotation process as painful, only 34% were satisfied with it, and 36% said more than half of their annotated data never reaches production. The industry's response has been twofold: annotate less data but the right data, and automate more of the labeling itself.

Annotation follows a workflow. Teams define a label schema and ontology, select which samples are worth labeling, label them through a mix of human and automated effort, review the labels through QA stages, and assemble the approved labels into training datasets. Errors found later, during model evaluation, come back as the next round's review targets. The diagram below summarizes the workflow.

Diagram of the data annotation workflow: schema and ontology, data selection, labeling, QA and review, and a labeled dataset, followed by training and evaluation. Rejected labels go from QA back to labeling, and model failures pick the next samples to label. — The annotation workflow: from schema design to labeled dataset, with rejection loops and evaluation feedback.

At a glance

Data annotation adds structured labels, such as bounding boxes, segmentation masks, 3D cuboids, and event intervals, to raw data so models can learn from it.
Annotation labels are ground truth. Their quality sets the ceiling on model quality, which makes QA a core part of annotation rather than an afterthought.
The annotation workflow runs from schema design through data selection, labeling, and review, with rejection loops and evaluation feedback built in.
Annotation automation now spans a spectrum from model-assisted labeling to auto-labeling with foundation models to agentic labeling driven by natural-language instructions.

Data annotation tutorials and guides from the Voxel51 blog

These posts cover annotation from schema design through QA and automation. Each section below links to more.

Best practices for producing high-quality training labels: the June 2026 Voxel51 Annotation labeling launch, covering workflows, ontologies, Agentic Labeling, and Intelligent Review.
A complete annotation workflow from unlabeled pool to fine-tuned detector: embed, deduplicate, prioritize, annotate, fine-tune, and correct, on a drone power-line dataset.
In-app annotation from a single sample to annotation at scale: schemas, brushes, polygons, and SAM 2 click-to-segment.
Curating data before annotation to cut labeling costs: coreset selection, 2D and 3D labeling, and error detection tied to evaluation.
Adding VLMs to your auto-labeling pipeline with Voxel51 Agentic Labeling: describe a labeling task in plain language and refine it until the labels match your guidelines.
Building an agentic labeling skill for an AI data agent: teach the agent to run Agentic Labeling on a cable-damage dataset.
Zero-shot auto-labeling rivals human performance: research on training detectors with labels from open-vocabulary models.
Pinpoint annotation issues with metrics and spatiotemporal comments: annotator metrics and reviewer comments pinned to a location or a frame.

Core concepts in data annotation

Annotation draws its vocabulary from labeling operations, quality measurement, and robot learning. Most terms are defined where they come up below. These six anchor the rest of this guide.

Ground truth: the labels accepted as correct, the reference a model is trained and measured against.
Annotation ontology: the structured set of labeled concepts and the relationships between them.
Annotation guidelines: the documented rules and examples annotators follow to label consistently.
Inter-annotator agreement: how consistently different annotators label the same data.
Human in the loop (HITL): people review and correct model outputs instead of leaving them fully automated.
Task annotation: the natural-language task descriptions attached to robot episodes, which language-conditioned policies learn to follow.

Label types in visual and physical AI

Every annotation task reduces to a small set of label types. The right one depends on what the model needs to learn: presence, location, shape, pose, or timing.

Label type	What it marks	Typical use
Classification	A whole image or sample, with no location	Scene tagging, quality flags, episode success or failure
Bounding box	Object location as a rectangle	Object detection
Polygon	Object outline	Irregular shapes and tight boundaries
Segmentation mask	The exact pixels belonging to an object or class	Semantic, instance, and panoptic segmentation
Keypoints	Labeled points and skeletons	Pose estimation, grasp points
Polyline	Open linear features	Lane lines, cables, and boundaries
3D cuboid	Position, size, and orientation in space	LiDAR and point cloud objects
Event	The start and end of an action in time	Video, robot subtasks such as reach, grasp, and place
Language	A natural-language description	Robot task instructions, captions for vision-language model (VLM) and vision-language-action (VLA) model training

Label types in visual and physical AI, with what each one marks and its typical use.

Editing an object label in Voxel51, correcting a misclassified annotation. — Correcting a label in place. Annotation quality is maintained by review.

How does the annotation workflow work?

A production annotation workflow is a pipeline, not a single act of drawing boxes. It starts before any labeling happens. The label schema defines what can be labeled and how, and annotation guidelines define the judgment calls, because two annotators given only a class list will disagree on every edge case. When a schema has to change mid-project, such as splitting one vehicle class into cars, trucks, and vans, mature teams version the ontology and remap or relabel older data so the training set stays consistent.

Selection comes next, and it is where annotation meets curation. Labeling everything is the default that wastes budgets. Active learning and model-assisted selection surface the samples most likely to improve the model, so annotation effort goes where it changes outcomes. In practice the selection signals are specific. Uncertainty sampling picks predictions where the model's confidence sits near the decision boundary, ensemble disagreement flags samples where several models give conflicting answers, and embedding coverage finds regions of the data the labeled set doesn't reach yet. In Voxel51, Smart Data Selection surfaces the samples most likely to improve the model so they get labeled first. The curation side of this story lives in the multimodal data guide.

Labeling itself runs through multi-stage pipelines: labeling, review, and approval stages with rejection loops that send substandard work back, and quality gates that keep unreviewed labels out of production datasets.

Annotation pipeline in Voxel51: a loaded dataset passes through an AI review step to a human review batch. — An annotation pipeline in Voxel51 routes a loaded dataset through AI review to a human review batch.

The automation spectrum: from manual labeling to agentic labeling

Annotation automation is not one thing. It is a spectrum, and most production teams operate at several points on it at once.

Manual labeling. A person draws every label. Still the standard for new tasks and the source of gold-standard reference data.

Model-assisted labeling. A model suggests labels in real time and a person accepts or corrects them. Click-to-segment is the common example: one click on an object produces a SAM 2 mask, and in video, tracking carries that mask through the clip while the annotator fixes the drift.

Auto-labeling. Also called automated labeling. Models, increasingly foundation models such as SAM 3 and open-vocabulary detectors, generate labels in bulk, and human effort moves to review. In research from Voxel51's ML team, object detectors trained only on labels from open-vocabulary detectors reached up to 95% of the performance of detectors trained on human labels on general-object benchmarks such as VOC and COCO, at up to 100,000x lower cost to generate the labels. Results were much weaker on rare and domain-specific classes, including the BDD driving benchmark, which is where human review still matters most.

Agentic labeling. The newest point on the spectrum, and a form of auto-labeling. A team describes the labeling task in natural language, a vision-language model applies it to a few samples, optionally with positive and negative example images, people review the results, and the instruction is refined like a prompt until outputs match the guidelines. Then it is applied across the dataset and saved for reuse across projects, which turns a team's labeling policy into a reusable, auditable artifact rather than knowledge that lives in individual annotators' heads. In Voxel51, this is Agentic Labeling.

The spectrum changes what annotators do rather than eliminating them. Human effort moves from drawing labels to defining tasks, reviewing outputs, and resolving edge cases. That shift is why QA is becoming the center of gravity of annotation.

Diagram of the annotation automation spectrum: manual labeling, model-assisted labeling, auto-labeling, and agentic labeling. A bar below shows people's time shifting from drawing every label to defining tasks and reviewing. — The annotation automation spectrum. Automation moves human effort toward review and edge cases rather than removing it.

SAM 2 video labeling in Voxel51, with a pedestrian labeled and tracked across video frames. — One click makes a SAM 2 mask, and tracking carries it through the clip.

Annotation QA: how teams keep labels trustworthy

Label noise, meaning incorrect, imprecise, or inconsistent labels mixed into ground truth, quietly degrades model performance, because a model trained on wrong answers learns them faithfully. Annotation QA is the set of practices that keeps noise out.

The traditional approach is spot-checking a random sample, which scales badly and misses systematic errors. Modern QA is model-assisted. Embeddings, model predictions, and comparisons against auto-labels surface the labels most likely to be wrong, so reviewers work through a ranked queue of high-risk labels instead of a random one, and systematic mistakes, such as a batch that mislabels one class, show up as clusters rather than anecdotes. Inter-annotator agreement measures consistency between people, and a gold set of expert-verified samples anchors what correct looks like. Annotator-level metrics, such as rejection rates and time per label, show where guidelines or training need work. In Voxel51, Intelligent Review ranks the labels most likely to be wrong so reviewers start there.

Not all disagreement is error. The hardest QA problems are real ambiguity: is a person walking a bicycle a pedestrian or a cyclist, and at what exact frame does a grasp begin? When agreement drops, the cause is often not careless annotators but guidelines that lack an edge-case policy, and research on commercial annotation teams found that better labeling instructions improved quality more than extra QA passes did. Modern QA treats disagreement as a signal with two possible fixes: fix the label, or fix the guideline that made the label ambiguous.

QA is also where annotation connects to everything else. As the 2026 survey report puts it, "The waste isn't an annotation problem. It's a data curation problem." Teams that select before they label, automate the volume, concentrate human judgment in review, and let evaluation failures pick the next round's targets treat annotation as part of the model development loop rather than a cost center in front of it.

Voxel51 surfacing a likely annotation error: a kiwi mislabeled as banana, found by ranking labels by mistakenness. — Model-assisted QA surfaces likely label errors, such as this kiwi labeled banana, ranked for review.

How are different modalities annotated?

The workflow holds across data types. What changes is the label geometry and the failure modes.

Image annotation

Images take the full range of 2D label types, from boxes to pixel-level masks. The image annotation tool in Voxel51 covers boxes, polygons, polylines, and SAM 2 masks.

Video annotation

Video annotation adds time. Labels become tracks that follow objects across frames, with interpolation and mask propagation filling in between keyframes, and events marking when actions start and end. The common failure shifts from one wrong frame to a track that drifts off its object over time. The video annotation tool in Voxel51 pairs interpolation with SAM 2 object tracking.

LiDAR and 3D point cloud annotation

LiDAR and 3D annotation replace boxes with cuboids and per-point classes on point clouds, with a constraint 2D annotation never faces: sparsity. A distant or occluded car may return only a handful of LiDAR points, and the annotator still has to draw a full, correctly oriented cuboid around it. The synchronized camera view of the same moment is often the only way to size that box, which is why multi-sensor annotation is a necessity in 3D rather than a convenience.

Robot episode annotation

Robot episodes are annotated mostly in language and time. Episodes in robot learning datasets carry a task description, and that string is the instruction a language-conditioned policy or VLA learns to follow, so a vague or wrong description teaches the wrong mapping from instruction to behavior. Teams add subtask and event labels with start and end times, such as reach, grasp, and place, plus success or failure flags, which makes failures easy to locate and supports training on long tasks.

Label	What it marks	Why policies need it
Task description	The instruction for the whole episode	Language-conditioned policies and VLAs learn the instruction-to-behavior mapping from it
Subtask events	Start and end of steps such as reach, grasp, and place	Long-horizon training, and locating where a failure starts
Success or failure flag	Whether the episode completed the task	Filtering training data and scoring rollouts
Object tracks or keypoints	Objects and grasp points over time	Perception models and policies that use object state

Robot episode labels, with what each one marks and why policies need it.

The LeRobot guide covers task annotation for LeRobot datasets.

Multimodal data annotation

Multimodal annotation labels synchronized streams together, and cross-modal projection keeps a label consistent across every sensor's view of the same moment. In the 2026 survey, 92% of teams work with images, 63% with video, and 38% with 3D point clouds, meshes, or LiDAR, and 68% work across three or more modalities. The multimodal data guide and the MCAP guide cover that world in more depth.

Label formats and standards

COCO is one of the most widely used formats for detection and segmentation labels, and ASAM OpenLABEL standardizes multi-sensor labels for automotive. LeRobot and RLDS carry task labels inside robot episodes.

Data annotation across physical AI

The workflow is the same across domains. What changes is the label geometry, the schema complexity, and where quality risk concentrates. The physical AI guide covers each domain in depth.

Robotics data annotation

In robotics, the risk is labeling the wrong episodes, so labeling starts with finding the right ones. Berkshire Grey's robotics team investigates picking failures three times faster with Voxel51.

Explore: robotics data platform for VLA models, Berkshire Grey customer story, how to collect teleoperation data for robot learning

Autonomous vehicle annotation

Advanced driver-assistance system (ADAS) and autonomous driving annotation schemas run to dozens of classes plus attributes, with severe imbalance. Even nuScenes, a public benchmark with 23 classes, has 493,322 car annotations and 49 ambulances in its trainval split, and the rare classes are often the safety-critical ones.

Explore: the long tail of autonomous driving: why rare cases beat more data, the long tail of autonomous driving (whitepaper)

Manufacturing and inspection annotation

Defect detection with extreme class imbalance, where the samples that matter most are the rarest.

Explore: visual AI for defect detection in manufacturing (whitepaper), ThreeV customer story

Defense annotation

Small targets, sensor-specific signatures in infrared, radar, and synthetic aperture radar (SAR), and annotators who need clearance make defense labeling different, and the data often can't leave an air-gapped network.

Explore: physical AI in defense

Agriculture annotation

Crops, weeds, fruit, and disease labeled across seasons, regions, and multispectral bands, where the same class can look completely different from one month to the next.

Explore: visual AI in agriculture: 2025 landscape

Try data annotation in your browser

You can run the labeling steps in this guide without writing code. In the free tier of Voxel51, you can:

Browse the 20+ demo datasets and add PandaSet, as 9,888 driving images or 103 driving clips. Each dataset, demo or your own, uses one of three free dataset slots.
Search for "cyclist at dusk," prompt "cyclists," and review the boxes the Voxel51 Agent drafts.
Click once on a pedestrian in a clip to make a SAM 2 mask, let tracking carry it to the end of the clip and back, and fix the drift. Frames inside a video don't count as separate samples.
Upload your own images or video and label them the same way.

Sign up with a work or .edu email, then start with the PandaSet images or clips. The tutorial on in-app annotation from a single sample to annotation at scale walks through schemas, brushes, and SAM 2 click-to-segment, and Try Voxel51 free covers what the free tier includes.

Data Annotation for Visual and Physical AI: Workflows, QA, and Auto-Labeling

Try Voxel51 for yourself