Multimodal data combines two or more distinct data types, such as camera video, LiDAR point clouds, inertial readings, and robot joint states, that describe the same subject or moment. This guide covers how multimodal data works in robotics and physical AI, the formats it lives in, and how teams replay, search, curate, annotate, and evaluate it at scale.
What is multimodal data in physical AI?
Physical AI systems, including robots, autonomous vehicles, and drones, perceive the world through many sensors at once. A single robot episode produces wrist and overhead camera frames, joint positions, gripper state, coordinate transforms, commanded actions, and often depth or force readings, all recorded as synchronized streams on a shared timeline. A single drive does the same with cameras, LiDAR, radar, GPS, and vehicle signals. Multimodal data is the collective name for these streams, and no single modality tells the full story on its own.
This guide is about multimodal sensor data used to build physical AI systems. That is related to, but distinct from, multimodal AI in the generative sense, where a model such as a vision-language model (VLM) accepts text and images as input. Here, multimodal means synchronized time-series streams from physical sensors, even when the model trained on them is itself a multimodal model, such as a vision-language-action (VLA) model.
Multimodal data exists because no single sensor captures the physical world completely, and the modality table below shows what each one adds. Physical AI combines these complementary signals through sensor fusion so a model perceives its environment more reliably than any individual sensor allows.
Working with multimodal data is harder than working with images or video alone for three reasons. First, the streams must stay synchronized, because a LiDAR sweep or a joint reading only means something when it is matched to the right camera frame. Second, the volume is enormous, since robot fleets and test vehicles record continuously. Third, the value is unevenly distributed. Most recorded data is routine, and the rare moments that improve a model are buried deep in the long tail.
Multimodal data moves through a lifecycle, from recording and replay through curation, annotation, training, and evaluation, and the diagram below summarizes it. The rest of this guide explores each stage.
Multimodal data in physical AI combines synchronized streams from cameras, LiDAR, radar, inertial measurement units (IMUs), and robot or vehicle state that describe the same moment.
Timing (synchronization) and space (calibration) determine whether multimodal sensor streams can be combined at all.
The multimodal data lifecycle runs from recording and replay through search, curation, annotation, training, and evaluation.
Model failures feed back into multimodal data curation, which makes dataset quality part of the model development loop.
Multimodal data tutorials and guides from the Voxel51 blog
These posts cover multimodal data from playback through search and curation. Each section below links to more.
The vocabulary of multimodal data comes from robotics, autonomous driving, and machine learning at once. These terms anchor the rest of this guide.
Multimodal data: two or more distinct data types describing the same subject or moment.
Multimodal sensor data: data captured at once from several sensor types describing the same scene.
Time-series sensor data: sensor readings recorded over time, where order and timing carry meaning.
Sensor fusion: combining data from multiple sensors into one estimate of the environment.
Sensor synchronization: aligning the timing of multiple sensors so readings match the same moment.
Sensor calibration: finding each sensor's internal parameters (intrinsics) and its position relative to the other sensors (extrinsics) so their data can be projected into a shared space.
Episode: one complete attempt at a task, recorded as a time-ordered sequence of observations and actions. Often called a trajectory or rollout. In robot learning, a demonstration is an episode performed by a person.
MCAP: the container file format for timestamped multimodal robotics data, and the Robot Operating System 2 (ROS 2) default.
LeRobot: Hugging Face's open source library and dataset format for robot learning.
Segment-level embedding: a vector computed over a short window inside a recording, so search can find a moment rather than a whole episode.
Sensor modalities in physical AI
Every physical AI system builds its view of the world from a different mix of sensors, and each modality has its own data shape, failure modes, and annotation needs.
Modality
What it captures
Why physical AI uses it
Cameras and video
Appearance, texture, and motion
High-resolution perception, and the main input to VLAs and most perception models
Robot state (proprioception)
Joint positions, velocities, forces, and gripper state
The robot's own sense of its body, and the observation a policy pairs with its actions
How is multimodal data stored? Formats and ingestion
Recording starts with synchronization. Each sensor's readings are timestamped against a shared clock, often using a protocol such as IEEE 1588 Precision Time Protocol (PTP) or a hardware trigger line, so every stream can be aligned to the same moments later. The tolerances are tight. If a robot's gripper moves at one meter per second, a 10-millisecond offset between the wrist camera and the joint readings is a centimeter of error, enough to make a good grasp look like a miss. At highway speed, the same offset between a camera and a LiDAR is about 30 centimeters of travel.
Time is only half of alignment. Sensor calibration handles space: intrinsics describe each sensor's internal optics, and extrinsics describe where each sensor sits relative to the others, so a LiDAR point can be projected onto the right camera pixel or a wrist camera can be located relative to the gripper. Calibration also drifts: a camera that gets bumped quietly corrupts every projection until someone notices.
The synchronized streams are then written into a container format that holds many timestamped streams in one file. MCAP is the ROS 2 default, and it is widely used outside ROS too. It stores every message a robot's sensors and software published, embeds the schemas needed to decode them, and became the default recording format in ROS 2 with the Iron release in 2023. Older ROS 2 recordings are often SQLite (.db3) files written by rosbag2 in Humble and earlier, and ROS 1 recordings live in rosbag (.bag) files. Both convert to MCAP. The MCAP guide covers the format in depth.
Robot learning datasets use their own formats. LeRobot and RLDS store episodes as ordered steps of observations and actions, structured for training rather than for logging. The LeRobot guide covers that format.
These formats are not alternatives to each other. They sit at different layers. Teams record to MCAP, curate the recordings, and then export the episodes worth training on into a dataset format such as LeRobot.
Format
Layer
Written
Structured around
Typical source
MCAP
Recording container
At capture, on the robot or vehicle
Timestamped message streams
ROS 2 systems and custom stacks
rosbag2 SQLite (.db3)
Recording container
At capture
Timestamped message streams
ROS 2 systems before Iron, such as Humble
rosbag (.bag)
Recording container
At capture
Timestamped message streams
Legacy ROS 1 systems
LeRobotDataset
Training dataset
At dataset assembly
Episodes of frames, state, and action
Imitation learning and VLA training
RLDS
Training dataset
At dataset assembly
Episodes as ordered steps
Open X-Embodiment and the research models trained on it
Multimodal robotics data formats, with the layer each sits at, when it's written, what it's structured around, and its typical source.
How do teams visualize and replay multimodal data?
Inspecting multimodal data one sensor at a time hides the information that matters most, which is how the streams relate. Replay, playing back recorded sensor streams exactly as the robot or vehicle experienced them, is how physical AI engineers inspect an episode after the fact. Synchronized playback puts camera frames, point clouds, and plots of joint state, actions, and IMU readings on one shared timeline, so an engineer can scrub to any moment and see the full context around a failure.
This is where multimodal context earns its keep. A failed grasp that looks inexplicable in the overhead camera can turn out to be a gripper that closed 200 milliseconds early, visible only in the gripper state plot.
How do teams search and curate multimodal datasets?
Replay answers questions about one episode. Curation answers questions about the whole dataset, such as how often a failure mode occurs, which scenarios are missing, and which episodes are worth labeling. Teams query multimodal datasets by metadata, temporal events, and labels, and use embeddings and semantic search to find visually similar moments that keyword filters miss. Longtail investigation, the practice of hunting the rare cases where models fail, is where curation pays off most. The episodes that curation surfaces flow downstream into annotation and training, which is why curation quality sets the ceiling on model quality.
The unit of search matters. An embedding computed over a whole episode can say that two episodes are similar overall, but a five-minute recording contains reaches, grasps, retries, and idle pauses, and the failure an engineer cares about may last two seconds. Segment-level embeddings are computed over short windows inside a recording, so a search for "gripper slips during lift" returns the moment it happens in every episode, not just the episodes that contain it somewhere.
Natural-language search changes the first step: instead of writing a filter, an engineer can ask for "episodes where the gripper drops the object" and review what comes back. Voxel51 indexes MCAP and LeRobot recordings this way, with natural-language, similarity, and segment-level search.
Episodes clustered by embedding similarity. Outliers are often the episodes worth watching first.
Annotating multimodal data and building training sets
Annotation for multimodal data goes beyond boxes on images. Labels exist as tracks that follow an object across a full episode, events are annotated as time intervals with start and end boundaries, such as reach, grasp, and place, and 3D cuboids place objects in space rather than on a flat frame. Because the modalities are synchronized, a label applied in one stream must stay consistent with every other stream at that moment, which makes multimodal annotation and quality assurance (QA) harder than single-modality labeling.
Cross-modal projection is how tooling keeps this honest. When an annotator draws a 3D cuboid around an object in the point cloud, the tool projects it onto the 2D camera views. If calibration and synchronization are correct, the projected box frames the object exactly. If it doesn't, the projection has just surfaced an alignment error.
Curated, labeled episodes then become training mixes that balance routine scenarios against the cases a model needs to see more of, such as failed grasps, rare objects, or night drives, with each mix versioned so every training run is reproducible. For robot learning, this step often ends in an export: curated MCAP recordings are cut into episodes, resampled to a common frame rate, and written out as a training format such as LeRobotDataset v3.0.
Aggregate metrics hide the failures that matter, and in multimodal systems the failures often live in one stream. A fused detector can score well overall and still miss every object that returns only a few LiDAR points, and a manipulation policy can succeed in good light and fail whenever the wrist camera is washed out, even though the joint data looks normal. Slicing evaluation by the condition of each modality, such as point density, exposure, sensor dropouts, and calibration session, alongside the usual slices by scenario and object class, is what surfaces these patterns.
Multimodal systems also face a failure mode single-sensor systems never do: sensor contradiction, where one modality reports an obstacle that the others don't. Finding the episodes where the streams disagree, and checking what the model did in each, is part of evaluating a fused system.
Evaluation closes the lifecycle. The scenarios that expose model failures become the next recordings engineers search for, annotate, and add to future training datasets.
The lifecycle is the same in every domain. What changes is the sensor mix, the shape of an episode, and where the long tail lives. The physical AI guide covers each domain in depth, along with the standards and formats they share.
Robotics
Robot episodes pair wrist and fixed cameras with joint states, actions, and often force or tactile readings, and every episode is slow to record.
You can run the replay and curation steps in this guide without writing code. In the free tier of Voxel51, you can:
Browse the 20+ demo datasets and add LeRobot Community, 2,378 robot episodes from 117 community datasets. Each dataset, demo or your own, uses one of three free dataset slots.
Play each episode's cameras on one synchronized timeline.
Plot the episodes by embedding similarity, open the outlier clusters, and use similarity search to pull every episode that looks like a bad one.
Ask the Voxel51 Agent to find episodes that match a plain-language description, and review what it finds.
Tag the moments that matter and save them as a view your team can reopen.
Sign up with a work or .edu email, then start with the LeRobot Community demo. The LeRobot support post shows the same playback, search, and tagging on LeRobot data, and Try Voxel51 free covers what the free tier includes. You can also upload your own MCAP and LeRobot files.
Multimodal data is the input: two or more data types describing the same thing. Multimodal AI is the class of models that reason across those types together, such as a VLA that connects camera frames and a language instruction to robot actions.
Multimodal data is the raw material, the synchronized streams recorded from multiple sensors. Sensor fusion is a technique applied to that data, combining the streams into a single, more reliable estimate of the environment. Every fusion system depends on the streams being synchronized and calibrated first.
MCAP is the default recording format in ROS 2, and it is widely used outside ROS for multimodal robotics data. It stores every sensor and software stream in one self-describing file, and older ROS 1 and ROS 2 bags convert to it. The MCAP guide compares the formats. For training, teams often convert curated recordings into an episode format such as LeRobotDataset or RLDS.
Sensor synchronization aligns the timing of multiple sensors so their readings can be matched to the same moment. Teams synchronize in hardware, where sensors share a clock or trigger signal at capture, or in software, where readings are matched afterward by nearest timestamp or by interpolating between them. Without it, a LiDAR sweep or joint reading cannot be paired with the camera frame it belongs to, which corrupts everything downstream.
Teams curate multimodal datasets by indexing recordings so they can be searched by metadata, temporal events, labels, and embedding similarity, then selecting the episodes that add coverage or fix known failure modes. Segment-level embeddings let search find a specific moment inside long recordings. Because most recorded data is routine, curation focuses on surfacing rare cases rather than collecting more volume.
Multimodal embeddings are vectors computed from one or more sensor streams, such as camera frames, point clouds, or a short window of a robot episode, so similar moments end up close together. Vision-language models such as CLIP and SigLIP map images and text into one shared space, which is what lets a plain-language query retrieve matching frames. Teams use embeddings for similarity search, finding coverage gaps, and removing near-duplicates. See a practical guide to multimodal embeddings for physical AI.