MCAP annotation is the labeling of robotics and sensor log data stored in the MCAP format, an open container for timestamped, multi-stream recordings, by marking objects and events across the synchronized sensor channels a robot records. It brings annotation to raw robot logs.
MCAP is an open container file format for storing heterogeneous, timestamped message streams, the kind of data a robot or autonomous system records: LiDAR, camera frames, IMU, GPS, and control signals, all on one synchronized timeline. It is widely used in the ROS 2 ecosystem, where it is the default storage format for rosbag2, the ROS 2 recording framework that succeeded ROS 1's rosbag. MCAP is a general-purpose format, not ROS-specific, and it is used outside robotics too.
MCAP annotation means labeling that log data: identifying objects and events across the recorded sensor channels and aligning labels to the shared timeline, so robot perception models can train on real operational recordings rather than curated image sets.
Key takeaways
MCAP is an open container format for timestamped, multi-stream robot and sensor logs.
MCAP annotation labels objects and events across those synchronized channels, on a shared timeline.
It is squarely physical AI: the raw, multimodal recordings robots actually produce.
What MCAP annotation provides
How MCAP relates to rosbag and rosbag2, and why they are not interchangeable.
How MCAP relates to rosbag and rosbag2, and why they are not interchangeable.
Term
What it is
Level
rosbag
The ROS 1 tool and .bag file format for recording message streams
ROS 1, end of life
rosbag2
The ROS 2 recording framework, storage-plugin based, not a single format
A tool, writes .mcap or .db3
MCAP
An open, general-purpose container format for timestamped multi-stream data
A format, the rosbag2 default, also used outside ROS
How it works
You label objects and events across the synchronized sensor channels in a recording, aligning each label to the shared timeline so it lines up across LiDAR, camera, and other streams. Because the recordings are long and continuous, the first step is finding the moments worth labeling, then annotating across the fused channels much like multimodal annotation. See the rosbag entry for the full rosbag, rosbag2, and MCAP distinction.
Why it matters
MCAP annotation is the labeling frontier for physical AI, because the data is the genuine article, full sensor recordings from real robots and vehicles, not a tidied dataset. The bottleneck is not labeling but finding what to label. A robot log is hours of mostly uneventful recording punctuated by the rare, valuable moments, a near-miss, an edge case, a failure, so the leverage is in curation, mining the logs for those moments, before any annotation happens. Treating an MCAP file like an image folder and labeling it uniformly wastes almost all the effort on data that teaches the model nothing.
Frequently asked questions
What is MCAP?
An open container file format for storing timestamped, multi-stream robotics and sensor data, used widely in ROS 2.
What is the difference between MCAP and rosbag2?
rosbag2 is the ROS 2 recording tool, MCAP is a file format it writes to. They are not alternatives at the same level, and MCAP is also usable entirely outside ROS.
What does MCAP annotation involve?
Labeling objects and events across the synchronized sensor channels in a robot log, aligned to a shared timeline.