Robotics Data Annotation: What to Label in Robot Data, How to Label It, and Which Tools Do It

Nov 18, 2026
•
11 min read
Robotics data annotation adds labels to robot recordings, such as camera video, LiDAR point clouds, joint states, and actions, so machine learning models can learn from them and be evaluated. The models are perception models that detect and track objects, and robot policies that learn to act from demonstrations. Robot annotation differs from image annotation in three ways: labels live on a timeline, several sensors share one clock, and many labels describe the task, such as the instruction, success or failure, and the phases of a grasp, rather than the objects in a frame.
This guide covers what gets labeled, how to label robot data in an order that saves money, and which tools do it. We make Voxel51. Every claim about another tool links to that vendor's own site or docs, checked November 13 to 18, 2026.

At a glance

  • Robotics data annotation uses two kinds of labels. Perception labels (boxes, masks, cuboids, keypoints) teach a model what is in the scene. Episode labels (instructions, success flags, phases, events) teach a robot policy what the robot was trying to do and whether it worked.
  • Robot data annotation saves money in one order: find, label, check. Labels on episodes that don't change the model are wasted spend, so find the episodes that matter before labeling them.
  • Robot labels belong on the recording's own timestamps. Cameras, LiDAR, and joint sensors log at different rates, so a label on a fixed frame grid drifts from the moment it describes.
  • Robotics data annotation tools split by job. Some label dense 3D on robot recordings, some label episodes in time, and some find the episodes worth labeling. Few do all three well.

What is robotics data annotation?

Robotics data annotation is the work of attaching labels to the data a robot records so that a model can learn from it or be measured against it. A recording, often called an episode, holds everything the robot logged during one attempt at a task: video from one or more cameras, depth or LiDAR, joint positions, gripper state, and the actions the robot or its teleoperator took.
Two kinds of models consume those labels.
  • Perception models learn to detect, segment, and track objects. They need spatial labels, such as boxes on camera frames or cuboids in a point cloud.
  • Robot policies, including imitation learning policies and vision-language-action models (VLAs), learn to map what the robot sees and is told to what it does. They need labels about the task: the instruction, whether the attempt succeeded, and where each phase began and ended.
Robot annotation differs from image annotation in three ways.
  1. Labels live on a timeline. A grasp, a slip, or a collision is an interval, not a frame.
  2. Several sensors share one clock. A cuboid in the point cloud and a box in the camera view describe the same object at the same moment, so they have to line up. This is multimodal annotation.
  3. Labels describe the task. The most useful label on a robot episode is often a single sentence and a pass or fail flag.
For the wider picture of how robots learn from data, see the physical AI guide.

What gets labeled in robot data?

Perception labels teach a model what is in the scene. Episode labels teach a policy what the robot tried, and whether it worked.
LabelWhat it's forTypical dataLearn more
2D boxes, masks, and tracksDetecting and tracking objects in camera viewsCamera videoVideo annotation
3D cuboids and tracksLocating objects in space, sized and orientedLiDAR and depth point clouds3D annotation
PolylinesLanes, edges, shelves, and other linear structuresPoint clouds, camera viewsPolyline annotation
Point cloud segmentationLabeling every point by classLiDARPoint cloud annotation
Keypoints and grasp pointsJoints, contact points, and where to grabCamera viewsKeypoint annotation
Task instructionThe sentence a language-conditioned policy learns to followWhole episodeTask annotation
Success or failureWhich demonstrations to train on, and how a policy didWhole episode
Phases or subtasksWhere reach, grasp, lift, and place begin and endTime rangesEvent annotation
EventsSlips, collisions, regrasps, and recoveriesTime rangesEvent annotation
Quality and exclusion flagsEpisodes or frames to leave out of trainingEpisodes, frames
Robot data labels, with what each is for, the data it goes on, and a glossary entry for each.
Perception labels are the ones most annotation tools were built for. Episode labels are what make robot data different. A policy trained on teleoperation demonstrations needs, at minimum, an instruction for each episode and a way to tell good demonstrations from bad ones. Long, multi-step tasks also benefit from phase labels, because they let you find and fix the step where a policy fails.

A worked example: one DROID episode

The DROID demo dataset in Voxel51 holds 1,000 robot manipulation episodes stored as MCAP recordings, with camera streams, robot telemetry, and labels at both levels. Each episode carries:
  • Episode labels: a task instruction plus up to two alternate phrasings of it, a success flag, a grasp count, and a short phase summary. This demo slice keeps only successful episodes, so every success flag is true.
  • Phase tags: time ranges on the recording's timeline for phases such as idle, reaching, grasping, transporting, releasing, and retreating, computed from the recorded joint and gripper streams when the dataset was prepared. In one 11-second episode, reaching runs from about 0.5 to 3.5 seconds, and a grasp of just over half a second follows it.
  • Events: moments flagged from the robot's own signals. Across the 1,000 episodes, the dataset lists 2,566 gripper state changes, 637 high end-effector speed events, and 200 high joint torque events.
With these labels you can ask questions a pile of demonstrations can't answer: which 228 episodes needed two or more grasps, how long the reach phase takes, or where joint torque spikes during transport. Computed labels like these are also a strong first pass for step 4 below: a model or a script proposes them, and people check the ones that matter.
One DROID episode: phase tags and events on the recording's timeline, under two of its camera views.

Why is robot data harder to annotate?

Sensors log at different rates. A camera might record 15 or 30 frames a second, a LiDAR spins 10 or 20 times a second, and joint encoders often report hundreds of times a second. Forcing all of them onto one frame grid either throws data away or creates frames no sensor recorded. Labels belong on the recording's own timestamps so they line up with the joint states and actions a policy trains on.
Snapping labels to a frame grid moves them. Labeling on the recording's own timestamps keeps them where the event happened. Rates are illustrative.
Episodes are long, and the moments that matter are short. A failed grasp can start and end in under a second inside a minute-long recording. Finding it by scrubbing video doesn't scale past a few dozen episodes.
The expensive mistake is labeling the wrong episodes. Teams often label everything they recorded, then discover the model needed more of the rare cases and fewer of the easy ones. Berkshire Grey's team made its robotics dataset investigations three times faster with Voxel51, turning tasks that took more than a day into quick, interactive filtering.

How do you annotate robot data?

Robot annotation works best in one order: make the recordings readable, find the episodes that matter, label them, check the labels, export them for training, and let the model's failures choose what to label next.
Where Voxel51 fits between recording a robot and training on its data.

1. Fix formats and timestamps first

Get every recording into a format your tools read, with timestamps you trust. MCAP has been the default rosbag2 storage format in the Robot Operating System (ROS 2) since the Iron release, and most tools in the table below read it. Some tools read ROS bags directly and some need them converted to MCAP first (the tools section below says which). For robot learning datasets, LeRobot is a common training format, alongside RLDS and HDF5 files. Check that cameras and joint states are actually in sync, and that camera intrinsics and extrinsics are recorded, before anyone draws a label. Without calibration, a cuboid in the point cloud can't be projected onto the camera views. A label placed on a misaligned frame teaches the model the wrong thing.

2. Find the episodes worth labeling

Search inside episodes for the moments you care about: failures, near misses, rare objects, and unusual conditions. Drop near-duplicate demonstrations that add cost without adding information. In Voxel51 you can search MCAP and LeRobot episodes in natural language on the free plan, using segment embeddings (beta). More on search and curation.

3. Write the schema and guidelines

Decide which labels you need, what each class means, and how to handle edge cases before labeling starts. A schema for a pick-and-place task might hold:
  • Episode: instruction (text), success (yes or no), quality (1 to 5)
  • Phases: reach, grasp, lift, place, each with a start and end time
  • Events: slip, regrasp, collision, recovery
  • Objects: the target object as a cuboid in the point cloud and a box in each camera view
For instructions, write a few phrasings of each task, as the DROID demo does with up to two alternates per episode, so a language-conditioned policy doesn't learn one wording. Write down what counts as the start of a grasp and when a slip becomes a failure. Most disagreements between labelers come from definitions nobody wrote down. See annotation ontology and annotation guidelines.

4. Pre-label with models

Let models do the first pass. Segmentation models such as SAM 2 can draw masks and track them through video, vision-language models can suggest instructions and event labels, and detectors trained on your own data can propose boxes and cuboids. Humans then correct instead of drawing from scratch. Voxel51's Agentic Labeling turns a plain-English description of a label into a reusable labeling agent.

5. Label in time and in 3D

Label phases and events as time ranges on the recording's timeline. Label objects in 3D as cuboid tracks in the point cloud, synced to the camera views so a sparse or distant object can be sized from the image. Track IDs should carry across frames and across sensors, so the same object keeps the same identity everywhere it appears.
In Voxel51, temporal tags mark time ranges on MCAP and LeRobot episodes. For 3D, version 2.26 added cuboid and polyline tracks that you label as a synchronized camera and LiDAR scene plays. Cuboid tracks interpolate between keyframes, and you can split and merge them on the timeline. These 3D labels work on grouped datasets of camera and LiDAR slices, not on MCAP or LeRobot episodes. The same release added keypoint and skeleton annotation. More on LiDAR annotation.

6. Review and measure quality

Check labels before they reach training. Hold back a gold set of episodes labeled by your most careful annotators, and score everyone else against it. Have two labelers mark the same episodes and measure how often they agree on success and on phase boundaries. Low agreement usually points to a guideline problem, not a labeler problem. In Voxel51, annotation projects can run benchmark batches that score labelers against ground truth and consensus batches where several labelers annotate the same samples. See inter-annotator agreement and annotation QA.

7. Export on the recording's timeline

Export labels tied to the recording's timestamps, in the format your training code reads. For robot policies that usually means LeRobot, where instructions and subtasks travel with the episodes. For perception models it means label files that reference the exact sensor message each label belongs to. For LeRobot data, Voxel51's Python SDK can export selected episodes of a LeRobot dataset as a standalone LeRobot v3 dataset. Temporal tags don't travel in that export. Voxel51's own dataset format keeps them.

8. Evaluate, then label what failed

Run the model, find where it fails, and send those episodes back to step 2. The cheapest labels are the ones aimed at a known failure. More in the model evaluation guide.
Find, label, and check as a loop: the model's failures decide what to label next.

Which tools label robot data?

No tool does every job best. The table below covers labeling software a team can run itself, with vendor docs showing either labels on ingested robot recordings or time-range labels on multi-camera robot episodes. Every cell links to the vendor's own source in the notes under the table. Beta features are marked for every vendor, including us. We don't rank tools.
This page compares labeling, so it includes tools that our robotics data tools guide lists under "other." That guide compares the whole data lifecycle, from recording to evaluation.
ToolRobot formats3D labelsTime and event labelsManaged labelingFree option
EncordMCAP (beta). Lists ROS 1 and ROS 2 bags as point cloud formatsCuboids, polylines, keypoints, and point cloud segmentation on MCAP (beta)Time ranges on numeric signals in MCAP (beta)YesNone listed
LightlyMCAP point-cloud sequencesCuboidsFrame tags on MCAP. No time ranges listedYesOpen source edition for one user and one dataset
Voxel51MCAP and LeRobot, plus LeRobot v3 export through the SDK. ROS bags convert to MCAP firstCuboid and polyline tracks on grouped camera and LiDAR datasets. Not on MCAP or LeRobot episodesTemporal tags on MCAP and LeRobot episodesNo, software onlyFree plan: 3 datasets, 10,000 samples each, 20 GB
AvalaMCAP. ROS 1 and ROS 2 bags convert on upload. LeRobot import through the SDKCuboids and polylinesEvent labels and classifications on frame rangesYesNone listed
Label StudioMulti-camera video episodes sent as JSON tasks. No robot recording formats listedCuboids on point clouds with keyframe tracks, in Enterprise and Starter CloudSubtask segments, mistake flags, and subgoals on a timeline, in Enterprise and Starter CloudYesOpen source Community Edition, without the robotics interface
Labeling tools for robot recordings, from each vendor's site and docs, checked November 18, 2026.
Encord labels MCAP recordings natively in beta, with every sensor on one timeline and labels exported on the recording's own timestamps through its software development kit (SDK). Access is through Encord's account team. Encord also sells data annotation and collection. Sources: MCAP in Encord, supported data, pricing.
Lightly added MCAP point-cloud sequences and 3D cuboid labeling to LightlyStudio in October 2026, and the v1.2.1 release lists work on writing cuboid labels as Foxglove SceneUpdate MCAP files and exporting MCAP sequences. Its assisted labeling uses SAM 3 through fal.ai. Lightly also sells annotation services. Sources: LightlyStudio changelog, v1.2.1 release, open source vs Enterprise, pricing.
Voxel51 reads MCAP and LeRobot natively, searches inside episodes with segment embeddings (beta), labels time ranges on them, and exports LeRobot datasets as LeRobot v3 through its Python SDK. Version 2.26 (released November 11, 2026) added multisensor temporal 3D annotation on grouped camera and LiDAR datasets, keypoint and skeleton annotation, and annotation projects with benchmark and consensus batches. MCAP indexing and segment embeddings are in beta. We make Voxel51. Sources: release notes, pricing (free plan limits), multimodal annotation.
Avala ingests MCAP and converts ROS 1 and ROS 2 bags on upload, labels cuboids, polylines, segmentation masks, keypoints, and events, and supplies managed labeling teams. Sources: supported formats, MCAP and ROS, robotics use case, avala.ai.
Label Studio from HumanSignal has a robotics episode review interface with a multi-camera video panel, a timeline for subtask segments, mistake flags, and subgoals, and a quality rating. A separate LiDAR interface labels cuboids on point clouds, reprojects them onto camera views, and edits them as keyframe tracks. Both interfaces run in the paid Enterprise and Starter Cloud editions. HumanSignal also sells labeling services, including physical AI and robotics data. Sources: robotics interfaces, LiDAR interface, edition comparison, labeling services.

3D and sensor fusion labeling tools

These tools label fused LiDAR and camera data, but their docs don't list robot recording formats such as MCAP, ROS bags, or LeRobot.
  • Segments.ai by Uber labels 3D cuboids, segmentation, keypoints, polygons, and polylines with track IDs across sensors, from PCD, KITTI, nuScenes, PLY, and LAS files, and offers labeling through partner agencies. Sources: multi-sensor labeling, supported file formats.
  • CVAT is open source and labels 3D point cloud cuboids with interpolation across sequences. Sources: CVAT robotics guide, formats.
  • Supervisely labels point cloud episodes imported from KITTI-360, nuScenes, Lyft, and point cloud files, and an ecosystem app exports point clouds and episodes to ROS bags. Sources: point cloud episodes, export to ROS bag.
  • Kognic labels fused camera, LiDAR, and radar data for autonomous driving, with OpenLABEL export, as a managed service or self-service. Source: Kognic platform.

Recording tools that mark events

Foxglove and Roboto mark events on time ranges in robot logs, and Rerun stores annotations alongside recordings. They are built for recording, replay, and log analysis rather than labeling, and our robotics data tools guide compares them by job.

Labeling services

If you'd rather hand the work to a team, Scale, Avala, Encord, Lightly, and Segments.ai partners all sell labeling for robot and sensor data. Scale also runs robotics data collection. See our roundup of data labeling services and companies.

Try Voxel51 with your robot data

Upload MCAP or LeRobot episodes to a free Voxel51 workspace, or explore 1,000 DROID episodes first. Search inside episodes and tag the moments that matter before you pay to label anything. No credit card, no time limit.

Try Voxel51 for yourself

Frequently asked questions about robotics data annotation

Have robot recordings to label? Try Voxel51 · See Voxel51 for robotics · Book a demo