Robotics Data Annotation: What to Label in Robot Data, How to Label It, and Which Tools Do It
Nov 18, 2026
• 11 min read
In this article
Try Voxel51 for yourself
Robotics data annotation adds labels to robot recordings, such as camera video, LiDAR point clouds, joint states, and actions, so machine learning models can learn from them and be evaluated. The models are perception models that detect and track objects, and robot policies that learn to act from demonstrations. Robot annotation differs from image annotation in three ways: labels live on a timeline, several sensors share one clock, and many labels describe the task, such as the instruction, success or failure, and the phases of a grasp, rather than the objects in a frame.
This guide covers what gets labeled, how to label robot data in an order that saves money, and which tools do it. We make Voxel51. Every claim about another tool links to that vendor's own site or docs, checked November 13 to 18, 2026.
At a glance
- Robotics data annotation uses two kinds of labels. Perception labels (boxes, masks, cuboids, keypoints) teach a model what is in the scene. Episode labels (instructions, success flags, phases, events) teach a robot policy what the robot was trying to do and whether it worked.
- Robot data annotation saves money in one order: find, label, check. Labels on episodes that don't change the model are wasted spend, so find the episodes that matter before labeling them.
- Robot labels belong on the recording's own timestamps. Cameras, LiDAR, and joint sensors log at different rates, so a label on a fixed frame grid drifts from the moment it describes.
- Robotics data annotation tools split by job. Some label dense 3D on robot recordings, some label episodes in time, and some find the episodes worth labeling. Few do all three well.
What is robotics data annotation?
Robotics data annotation is the work of attaching labels to the data a robot records so that a model can learn from it or be measured against it. A recording, often called an episode, holds everything the robot logged during one attempt at a task: video from one or more cameras, depth or LiDAR, joint positions, gripper state, and the actions the robot or its teleoperator took.
Two kinds of models consume those labels.
- Perception models learn to detect, segment, and track objects. They need spatial labels, such as boxes on camera frames or cuboids in a point cloud.
- Robot policies, including imitation learning policies and vision-language-action models (VLAs), learn to map what the robot sees and is told to what it does. They need labels about the task: the instruction, whether the attempt succeeded, and where each phase began and ended.
Robot annotation differs from image annotation in three ways.
- Labels live on a timeline. A grasp, a slip, or a collision is an interval, not a frame.
- Several sensors share one clock. A cuboid in the point cloud and a box in the camera view describe the same object at the same moment, so they have to line up. This is multimodal annotation.
- Labels describe the task. The most useful label on a robot episode is often a single sentence and a pass or fail flag.
For the wider picture of how robots learn from data, see the physical AI guide.
What gets labeled in robot data?
| Label | What it's for | Typical data | Learn more |
|---|---|---|---|
| 2D boxes, masks, and tracks | Detecting and tracking objects in camera views | Camera video | Video annotation |
| 3D cuboids and tracks | Locating objects in space, sized and oriented | LiDAR and depth point clouds | 3D annotation |
| Polylines | Lanes, edges, shelves, and other linear structures | Point clouds, camera views | Polyline annotation |
| Point cloud segmentation | Labeling every point by class | LiDAR | Point cloud annotation |
| Keypoints and grasp points | Joints, contact points, and where to grab | Camera views | Keypoint annotation |
| Task instruction | The sentence a language-conditioned policy learns to follow | Whole episode | Task annotation |
| Success or failure | Which demonstrations to train on, and how a policy did | Whole episode | |
| Phases or subtasks | Where reach, grasp, lift, and place begin and end | Time ranges | Event annotation |
| Events | Slips, collisions, regrasps, and recoveries | Time ranges | Event annotation |
| Quality and exclusion flags | Episodes or frames to leave out of training | Episodes, frames |
Perception labels are the ones most annotation tools were built for. Episode labels are what make robot data different. A policy trained on teleoperation demonstrations needs, at minimum, an instruction for each episode and a way to tell good demonstrations from bad ones. Long, multi-step tasks also benefit from phase labels, because they let you find and fix the step where a policy fails.
A worked example: one DROID episode
The DROID demo dataset in Voxel51 holds 1,000 robot manipulation episodes stored as MCAP recordings, with camera streams, robot telemetry, and labels at both levels. Each episode carries:
- Episode labels: a task instruction plus up to two alternate phrasings of it, a success flag, a grasp count, and a short phase summary. This demo slice keeps only successful episodes, so every success flag is true.
- Phase tags: time ranges on the recording's timeline for phases such as idle, reaching, grasping, transporting, releasing, and retreating, computed from the recorded joint and gripper streams when the dataset was prepared. In one 11-second episode, reaching runs from about 0.5 to 3.5 seconds, and a grasp of just over half a second follows it.
- Events: moments flagged from the robot's own signals. Across the 1,000 episodes, the dataset lists 2,566 gripper state changes, 637 high end-effector speed events, and 200 high joint torque events.
With these labels you can ask questions a pile of demonstrations can't answer: which 228 episodes needed two or more grasps, how long the reach phase takes, or where joint torque spikes during transport. Computed labels like these are also a strong first pass for step 4 below: a model or a script proposes them, and people check the ones that matter.
Why is robot data harder to annotate?
Sensors log at different rates. A camera might record 15 or 30 frames a second, a LiDAR spins 10 or 20 times a second, and joint encoders often report hundreds of times a second. Forcing all of them onto one frame grid either throws data away or creates frames no sensor recorded. Labels belong on the recording's own timestamps so they line up with the joint states and actions a policy trains on.
Episodes are long, and the moments that matter are short. A failed grasp can start and end in under a second inside a minute-long recording. Finding it by scrubbing video doesn't scale past a few dozen episodes.
The expensive mistake is labeling the wrong episodes. Teams often label everything they recorded, then discover the model needed more of the rare cases and fewer of the easy ones. Berkshire Grey's team made its robotics dataset investigations three times faster with Voxel51, turning tasks that took more than a day into quick, interactive filtering.
How do you annotate robot data?
Robot annotation works best in one order: make the recordings readable, find the episodes that matter, label them, check the labels, export them for training, and let the model's failures choose what to label next.
1. Fix formats and timestamps first
Get every recording into a format your tools read, with timestamps you trust. MCAP has been the default rosbag2 storage format in the Robot Operating System (ROS 2) since the Iron release, and most tools in the table below read it. Some tools read ROS bags directly and some need them converted to MCAP first (the tools section below says which). For robot learning datasets, LeRobot is a common training format, alongside RLDS and HDF5 files. Check that cameras and joint states are actually in sync, and that camera intrinsics and extrinsics are recorded, before anyone draws a label. Without calibration, a cuboid in the point cloud can't be projected onto the camera views. A label placed on a misaligned frame teaches the model the wrong thing.
2. Find the episodes worth labeling
Search inside episodes for the moments you care about: failures, near misses, rare objects, and unusual conditions. Drop near-duplicate demonstrations that add cost without adding information. In Voxel51 you can search MCAP and LeRobot episodes in natural language on the free plan, using segment embeddings (beta). More on search and curation.
3. Write the schema and guidelines
Decide which labels you need, what each class means, and how to handle edge cases before labeling starts. A schema for a pick-and-place task might hold:
- Episode: instruction (text), success (yes or no), quality (1 to 5)
- Phases: reach, grasp, lift, place, each with a start and end time
- Events: slip, regrasp, collision, recovery
- Objects: the target object as a cuboid in the point cloud and a box in each camera view
For instructions, write a few phrasings of each task, as the DROID demo does with up to two alternates per episode, so a language-conditioned policy doesn't learn one wording. Write down what counts as the start of a grasp and when a slip becomes a failure. Most disagreements between labelers come from definitions nobody wrote down. See annotation ontology and annotation guidelines.
4. Pre-label with models
Let models do the first pass. Segmentation models such as SAM 2 can draw masks and track them through video, vision-language models can suggest instructions and event labels, and detectors trained on your own data can propose boxes and cuboids. Humans then correct instead of drawing from scratch. Voxel51's Agentic Labeling turns a plain-English description of a label into a reusable labeling agent.
5. Label in time and in 3D
Label phases and events as time ranges on the recording's timeline. Label objects in 3D as cuboid tracks in the point cloud, synced to the camera views so a sparse or distant object can be sized from the image. Track IDs should carry across frames and across sensors, so the same object keeps the same identity everywhere it appears.
In Voxel51, temporal tags mark time ranges on MCAP and LeRobot episodes. For 3D, version 2.26 added cuboid and polyline tracks that you label as a synchronized camera and LiDAR scene plays. Cuboid tracks interpolate between keyframes, and you can split and merge them on the timeline. These 3D labels work on grouped datasets of camera and LiDAR slices, not on MCAP or LeRobot episodes. The same release added keypoint and skeleton annotation. More on LiDAR annotation.
6. Review and measure quality
Check labels before they reach training. Hold back a gold set of episodes labeled by your most careful annotators, and score everyone else against it. Have two labelers mark the same episodes and measure how often they agree on success and on phase boundaries. Low agreement usually points to a guideline problem, not a labeler problem. In Voxel51, annotation projects can run benchmark batches that score labelers against ground truth and consensus batches where several labelers annotate the same samples. See inter-annotator agreement and annotation QA.
7. Export on the recording's timeline
Export labels tied to the recording's timestamps, in the format your training code reads. For robot policies that usually means LeRobot, where instructions and subtasks travel with the episodes. For perception models it means label files that reference the exact sensor message each label belongs to. For LeRobot data, Voxel51's Python SDK can export selected episodes of a LeRobot dataset as a standalone LeRobot v3 dataset. Temporal tags don't travel in that export. Voxel51's own dataset format keeps them.
8. Evaluate, then label what failed
Run the model, find where it fails, and send those episodes back to step 2. The cheapest labels are the ones aimed at a known failure. More in the model evaluation guide.
Which tools label robot data?
No tool does every job best. The table below covers labeling software a team can run itself, with vendor docs showing either labels on ingested robot recordings or time-range labels on multi-camera robot episodes. Every cell links to the vendor's own source in the notes under the table. Beta features are marked for every vendor, including us. We don't rank tools.
This page compares labeling, so it includes tools that our robotics data tools guide lists under "other." That guide compares the whole data lifecycle, from recording to evaluation.
| Tool | Robot formats | 3D labels | Time and event labels | Managed labeling | Free option |
|---|---|---|---|---|---|
| Encord | MCAP (beta). Lists ROS 1 and ROS 2 bags as point cloud formats | Cuboids, polylines, keypoints, and point cloud segmentation on MCAP (beta) | Time ranges on numeric signals in MCAP (beta) | Yes | None listed |
| Lightly | MCAP point-cloud sequences | Cuboids | Frame tags on MCAP. No time ranges listed | Yes | Open source edition for one user and one dataset |
| Voxel51 | MCAP and LeRobot, plus LeRobot v3 export through the SDK. ROS bags convert to MCAP first | Cuboid and polyline tracks on grouped camera and LiDAR datasets. Not on MCAP or LeRobot episodes | Temporal tags on MCAP and LeRobot episodes | No, software only | Free plan: 3 datasets, 10,000 samples each, 20 GB |
| Avala | MCAP. ROS 1 and ROS 2 bags convert on upload. LeRobot import through the SDK | Cuboids and polylines | Event labels and classifications on frame ranges | Yes | None listed |
| Label Studio | Multi-camera video episodes sent as JSON tasks. No robot recording formats listed | Cuboids on point clouds with keyframe tracks, in Enterprise and Starter Cloud | Subtask segments, mistake flags, and subgoals on a timeline, in Enterprise and Starter Cloud | Yes | Open source Community Edition, without the robotics interface |
Encord labels MCAP recordings natively in beta, with every sensor on one timeline and labels exported on the recording's own timestamps through its software development kit (SDK). Access is through Encord's account team. Encord also sells data annotation and collection. Sources: MCAP in Encord, supported data, pricing.
Lightly added MCAP point-cloud sequences and 3D cuboid labeling to LightlyStudio in October 2026, and the v1.2.1 release lists work on writing cuboid labels as Foxglove SceneUpdate MCAP files and exporting MCAP sequences. Its assisted labeling uses SAM 3 through fal.ai. Lightly also sells annotation services. Sources: LightlyStudio changelog, v1.2.1 release, open source vs Enterprise, pricing.
Voxel51 reads MCAP and LeRobot natively, searches inside episodes with segment embeddings (beta), labels time ranges on them, and exports LeRobot datasets as LeRobot v3 through its Python SDK. Version 2.26 (released November 11, 2026) added multisensor temporal 3D annotation on grouped camera and LiDAR datasets, keypoint and skeleton annotation, and annotation projects with benchmark and consensus batches. MCAP indexing and segment embeddings are in beta. We make Voxel51. Sources: release notes, pricing (free plan limits), multimodal annotation.
Avala ingests MCAP and converts ROS 1 and ROS 2 bags on upload, labels cuboids, polylines, segmentation masks, keypoints, and events, and supplies managed labeling teams. Sources: supported formats, MCAP and ROS, robotics use case, avala.ai.
Label Studio from HumanSignal has a robotics episode review interface with a multi-camera video panel, a timeline for subtask segments, mistake flags, and subgoals, and a quality rating. A separate LiDAR interface labels cuboids on point clouds, reprojects them onto camera views, and edits them as keyframe tracks. Both interfaces run in the paid Enterprise and Starter Cloud editions. HumanSignal also sells labeling services, including physical AI and robotics data. Sources: robotics interfaces, LiDAR interface, edition comparison, labeling services.
3D and sensor fusion labeling tools
These tools label fused LiDAR and camera data, but their docs don't list robot recording formats such as MCAP, ROS bags, or LeRobot.
- Segments.ai by Uber labels 3D cuboids, segmentation, keypoints, polygons, and polylines with track IDs across sensors, from PCD, KITTI, nuScenes, PLY, and LAS files, and offers labeling through partner agencies. Sources: multi-sensor labeling, supported file formats.
- CVAT is open source and labels 3D point cloud cuboids with interpolation across sequences. Sources: CVAT robotics guide, formats.
- Supervisely labels point cloud episodes imported from KITTI-360, nuScenes, Lyft, and point cloud files, and an ecosystem app exports point clouds and episodes to ROS bags. Sources: point cloud episodes, export to ROS bag.
- Kognic labels fused camera, LiDAR, and radar data for autonomous driving, with OpenLABEL export, as a managed service or self-service. Source: Kognic platform.
Recording tools that mark events
Foxglove and Roboto mark events on time ranges in robot logs, and Rerun stores annotations alongside recordings. They are built for recording, replay, and log analysis rather than labeling, and our robotics data tools guide compares them by job.
Labeling services
If you'd rather hand the work to a team, Scale, Avala, Encord, Lightly, and Segments.ai partners all sell labeling for robot and sensor data. Scale also runs robotics data collection. See our roundup of data labeling services and companies.
Try Voxel51 with your robot data
Upload MCAP or LeRobot episodes to a free Voxel51 workspace, or explore 1,000 DROID episodes first. Search inside episodes and tag the moments that matter before you pay to label anything. No credit card, no time limit.
Try Voxel51 for yourself
Frequently asked questions about robotics data annotation
Robotics data annotation adds labels to robot recordings, such as camera video, LiDAR point clouds, joint states, and actions, so perception models and robot policies can learn from them. It includes spatial labels, such as boxes and cuboids, and episode labels, such as the task instruction, success or failure, and the start and end of each phase.
A robot policy needs, at minimum, a task instruction and a success or failure flag for each episode. Policies trained on long, multi-step tasks also benefit from phase or subtask labels with start and end times, which make it possible to find the step where a policy fails. Perception labels such as boxes and cuboids matter more for the models that feed a policy than for the policy itself.
To annotate MCAP files, load each recording into a tool that reads MCAP natively, so every sensor plays on the recording's own timeline. Label time ranges for phases and events, and label objects in 3D where your tool supports it. Encord labels MCAP in beta, Lightly labels cuboids on MCAP point-cloud sequences, Avala ingests MCAP, and Voxel51 reads MCAP natively and tags time ranges on episodes. Export labels tied to the recording's timestamps. More in the MCAP guide.
Avala and Encord accept ROS bags. Avala converts ROS 1 and ROS 2 bags on upload, and Encord lists ROS 1 and ROS 2 bags as point cloud formats. Voxel51 reads MCAP, so ROS bags are converted to MCAP first. MCAP has been the default ROS 2 storage format since the Iron release, so most new recordings won't need converting.
LeRobot datasets already carry a task instruction for each episode. To add phases, subtasks, or events, label time ranges on each episode and write them back into the dataset so they travel to training. Hugging Face's lerobot-annotate runs a vision-language model over each episode of a LeRobotDataset v3.0 and writes subtasks, plans, and other language annotations into the dataset's language columns, where a policy can train on them. Voxel51 reads LeRobot datasets, tags time ranges on each episode, and its Python SDK exports episodes as LeRobot v3 datasets. Temporal tags don't travel in that export. They stay in Voxel51's own dataset format. More in the LeRobot guide.
To label teleoperation data for VLA training, give every demonstration a clear, specific instruction, flag failed or low-quality demonstrations so they can be filtered out, and label phases on long tasks. Check that the instruction matches what the operator actually did, because a vague or wrong instruction teaches the policy the wrong mapping. More on teleoperation data collection and VLA fine-tuning.
The best robotics data annotation tool depends on the job. For dense 3D labels on MCAP recordings, Encord (in beta), Lightly (cuboids), and Avala document native support. For reviewing multi-camera episodes on a timeline, Label Studio has a dedicated interface in its paid editions. For adding language labels to LeRobot datasets with a model, Hugging Face has lerobot-annotate. For finding the episodes worth labeling and tagging time ranges on them, Voxel51 reads MCAP and LeRobot natively and has a free plan. Its 3D labeling works on grouped camera and LiDAR datasets.
Outsourcing robot data labeling makes sense for large volumes of well-defined labels, such as cuboids on thousands of LiDAR sweeps. Keep episode labels, such as success flags and phase boundaries, close to the team that understands the task, at least until the guidelines are stable. Either way, curate first so you pay to label the episodes that change the model. See our roundup of data labeling services and companies.