A field guide to 10 robotics datasets (driving, aquatic, forest, indoor, and synthetic) repackaged as time-synchronized MCAP recordings for FiftyOne’s native multimodal viewer, with the exact sensor suite, label situation, and a direct link to the paper, the dataset, and (where one exists) a live FiftyOne Space for each.
A camera on a driving rig fires at 10-20 Hz, its LiDAR spins at 10 Hz, its inertial measurement unit (IMU) ticks at 200-400 Hz, and its GPS/INS updates entirely on its own clock. None of these streams line up into a clean “one image = one label = one sample” unit, and yet that unit is what almost every computer vision tool assumes you have.
Robot data doesn’t work that way. A single recording is a bundle of asynchronous, multi-rate streams captured over minutes, not a folder of independent frames. Trying to force it into per-frame samples either throws away the temporal relationship between sensors or multiplies your sample count by however many topics you’re logging.
That’s the problem MCAP was built to solve, and it’s why FiftyOne added native multimodal support for it. We pulled together 10 datasets, all repackaged as MCAP episodes on the Voxel51 Hugging Face org, spanning urban driving, handheld SLAM, aquatic navigation, off-road forest robotics, indoor human-robot interaction, and one fully synthetic set. Below: what MCAP actually is, the data curation problems that show up over and over when repackaging real robot logs into it, a full comparison and schema reference, and a domain-by-domain tour of all 10 datasets with links to the paper, the dataset, and the FiftyOne Space where one exists.
Key Takeaways
10 datasets, 89 total episodes, spanning ground-vehicle driving, campus/handheld SLAM, aquatic navigation, off-road forest robotics, indoor social navigation, and one synthetic set (TartanGround). Every one is a FiftyOne multimodal dataset, where each sample is a continuous recording, not a single frame.
Only 3 of 10 carry real, embedded object-level labels: SemanticSpray++ (all 36 episodes fully labeled, 2D camera boxes, 3D LiDAR boxes, LiDAR + radar point-wise semantics), boreas-multimodal (only 6 of its 8 episodes, the boreas-objects-v1 windows, not the 2 driving episodes), and cmht-autonomous-driving (3D + derived 2D detections on all 4 drives). 6 more (yuto-mms-multimodal, oxford-spires-multimodal, canoe-multimodal, fomo-multimodal-sample, treescope-vat0723-multimodal, TartanGround) are raw SLAM/odometry sensor data with zero object annotations.
navwareset sits in between. Pedestrian 3D cuboids exist and are real, but they’re logged as foxglove.SceneUpdate entities inside the MCAP timeline, not a FiftyOne label field, the same modeling choice used for Boreas’s and CMHT’s boxes, since a multi-minute episode has no single fixed-length list a sample-level label field could hold.
Sensor suites go well beyond camera + LiDAR: boreas-multimodal, canoe-multimodal, and fomo-multimodal-sample all carry radar; canoe-multimodal adds imaging sonar; cmht-autonomous-driving adds a thermal/IR camera; fomo-multimodal-sample adds stereo audio. Three datasets (yuto-mms-multimodal, oxford-spires-multimodal, boreas-multimodal) ship survey-grade ground-truth trajectories accurate to 1-4 cm.
Licensing is genuinely mixed: CC BY 4.0 is most common, but oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA (non-commercial), cmht-autonomous-driving is CC0, boreas-multimodal’s license is unlisted/unknown, and yuto-mms-multimodal has a genuine conflict: the project site says CC BY-NC-SA 4.0, the live Zenodo records serve CC BY 4.0.
4 of 10 have no arXiv preprint at all (yuto-mms-multimodal, navwareset, canoe-multimodal, cmht-autonomous-driving), DOI/journal citation only, so don’t go hunting for a paper ID that doesn’t exist.
What is MCAP, and why did it become the physical AI format?
MCAP (pronounced “em-cap”) is a modular, serialization-agnostic binary container format for timestamped pub/sub data. A single .mcap file holds any number of named channels (camera frames, LiDAR scans, IMU samples, GPS fixes, transform trees), each tagged with its own schema, all indexed together so a reader can seek to any timestamp without scanning the whole file. Foxglove built it in 2022 after noticing that robotics teams kept burning engineering time reinventing custom binary formats, only to make third-party tooling harder later. It’s now the default log format for ROS 2 (since the Iron release in May 2023), replacing the SQLite3-backed .db3 bags, which struggled with high-throughput sequential writes.
Serialization-agnostic. One file can hold ROS 1 messages, ROS 2 messages, Protobuf, JSON Schema, and FlatBuffers side by side. That’s exactly the mix you get on a real rig with a ROS-based LiDAR driver, a vendor SDK camera, and a custom logger.
Self-contained. Message schemas are embedded in the file itself, so an MCAP recorded years ago is still readable without the original robot software stack, a real concern for datasets that are already 1-5 years old.
Append-only writing. Data streams to disk without ever seeking backward, so a recording survives a software crash or power loss mid-capture rather than being corrupted.
Chunk-indexed for fast seeking. A chunk index at the end of the file lets a reader jump directly to a timestamp instead of scanning from the start. That’s the specific mechanic that makes scrubbing a 20-minute episode in FiftyOne’s multimodal viewer responsive instead of painful.
Common data curation issues when repackaging real robot logs into MCAP
Each of the 10 datasets below started life as raw ROS bags, per-file sensor dumps, or vendor SDK output and had to be converted. The same handful of problems showed up repeatedly across completely unrelated source datasets, worth knowing before you repackage your own robot logs.
No native MCAP schema for polar or non-camera sensors. Radar (boreas-multimodal, canoe-multimodal, fomo-multimodal-sample) and imaging sonar (canoe-multimodal) have no dedicated Foxglove schema. All three decode their radar into a bird’s-eye-viewfoxglove.RawImage pseudo-image rather than a real camera view, the closest available representation, not a native one.
Broken or placeholder timestamps that require rebasing; not trusting.treescope-vat0723-multimodal’s processed bag stamped every message with the wall-clock time of an offline reprocessing job run weeks after the actual flight, not the flight’s capture time. Left uncorrected, an episode would nominally span that entire gap with real data in only two tiny slivers. cmht-autonomous-driving’s LiDAR and camera topics had a placeholder sec=0 header on every raw message. navwareset’s robot /tf carried real but non-physical roll/pitch/z noise, verified against the same bag’s adaptive Monte Carlo localization (AMCL) pose (which was exactly zero on every message) before flattening it.
Frame-count assumptions that quietly break on real data.cmht-autonomous-driving’s labels are published against a separately numbered “extracted frame” release, and an initial assumption of “labeled frame N = the Nth bag message” held for exactly 1 of 4 drives. The other 3 had extra bag messages that the extraction dropped, growing throughout the drive. The fix required content-based matching (pixel equality for images, near-exact point-coordinate equality for LiDAR), not positional counting.
Missing calibration that has to be derived, never fabricated.semanticspray-plusplus ships with zero camera calibration anywhere in the source release, the devkit, or the community-supplied calibration file, so its camera stream is intentionally left unconnected to the transform tree rather than given an invented pose. oxford-spires-multimodal’s depth/normal-map pipeline needs open3d, which has no wheel for the build machine’s platform, so the devkit’s own math was reimplemented in NumPy/OpenCV instead of skipped.
Placeholder or leaked values that look like real per-point data but aren’t.canoe-multimodal’s LiDAR preallocates a 131,072-point array per frame; 70-93% of those points are (0,0,0) “no return” filler that has to be filtered out, or the 3D tile shows a dense blob at the origin. semanticspray-plusplus’s radar z-column is a dataset-wide constant that’s exactly the sensor’s calibrated mount height, leaked from upstream extraction rather than a real per-point measurement, so height is set via a static transform instead.
Which MCAP dataset should you use?
All 10 MCAP datasets side by side: domain, episode count, sensor suite, whether real object labels are embedded, and license.
All 10 MCAP datasets side by side: domain, episode count, sensor suite, whether real object labels are embedded, and license.
Dataset
Domain
Episodes
Sensor Suite
Object Labels?
License
YUTO MMS
Ground-vehicle mobile mapping (SLAM)
2
Panoramic camera (Ladybug 5), tilted 32-beam LiDAR, GNSS/INS, IMU
Pedestrian 3D cuboids (SceneUpdate, not a label field)
CC BY 4.0
Keep it honest. “Episodes” reflects each FiftyOne build’s current scope, not the full source dataset. Every one of these is a partial repackaging (see each domain section for exactly how much of the source is covered and why).
What does "episode" mean in FiftyOne?
For every dataset below, dataset.media_type == "multimodal" and one sample is one episode, one continuous recording, stored as a single .mcap file. There’s no per-frame image or point-cloud sample sitting alongside it; every camera frame, LiDAR scan, IMU sample, and transform inside that episode is decoded live by FiftyOne’s multimodal viewer, which renders Image, 3D, Map, Plot, Logs, and Message tiles in sync as you scrub through the recording:
Object-level labels, where they exist, follow the same logic: a 3D box that’s only valid for one instant inside a 10-minute recording can’t live in a fixed-length sample field, so it’s logged as another timestamped channel inside the same MCAP (foxglove.SceneUpdate for 3D cuboids, foxglove.ImageAnnotations for 2D overlays), decoded in sync with the sensor data rather than stored separately.
Dataset schema reference: media types, sample fields, and links
What each dataset looks like once loaded in FiftyOne: media type, the sample fields you can filter on, and where labels actually live.
What each dataset looks like once loaded in FiftyOne: media type, the sample fields you can filter on, and where labels actually live.
Ground-vehicle and campus SLAM datasets (raw, no object labels)
Both of these are raw multi-sensor recordings built for SLAM/odometry/reconstruction benchmarking, not perception. Neither carries a single object-level annotation. Their “ground truth” is a survey-grade trajectory, not a bounding box.
YUTO MMS
YUTO MMS (York University Teledyne Optech Mobile Mapping System) is a SLAM benchmark collected with Teledyne Optech’s Maverick mobile mapping unit: a 360° Ladybug 5 panoramic camera (six stitched CCD sensors, 8000×4000), a Velodyne HDL-32E LiDAR tilted 45° off the camera axis, and a NovAtel GNSS/INS, recorded across a controlled parking-lot sequence in Vaughan, Ontario and a residential-loop sequence around York University’s Keele Campus in Toronto. This build covers 2 of the source’s 4 sequences (A and D); B and C are still in progress, since each requires re-tuning the accumulated-map voxel size and update rate per sequence rather than reusing a previous one’s settings. Every LiDAR point is colorized with real RGB sampled from the nearest-in-time panorama via the dataset’s own boresight calibration, not a synthetic colormap.
Fields.sequence_id, route_length_m, collection_site, duration_s, plus has_image/has_pointcloud/has_gps/has_imu capability flags. No label fields exist, and none should. The source paper’s entire “Dataset” section covers only raw sensor streams and a photogrammetric ground-truth trajectory.
Keep it honest. The license is genuinely conflicting: the project site and README both state CC BY-NC-SA 4.0, but every live Zenodo record for every sequence checked directly serves CC BY 4.0 via its own rel="license" header. Verify current status before redistributing.
Oxford Spires is the Oxford Robotics Institute’s large-scale LiDAR-visual localization, reconstruction, and radiance-field benchmark, captured by walking a handheld backpack unit called Frontier: three synchronized fisheye cameras (forward/left/right), a 64-beam LiDAR, and an IMU, through six historic Oxford landmarks (Bodleian Library, Christ Church, Keble College, New College, Blenheim Palace, Radcliffe Observatory Quarter). Each site also has a millimeter-accurate Terrestrial LiDAR Scanner reference model, used to produce centimeter-accurate ground-truth trajectories via iterative closest point (ICP) registration. Six of the source’s 24 sequences make it into this build, one per site, choosing the smallest available sequence at each landmark. Alongside the raw streams, every episode also carries the source devkit’s own derived depth maps, HSV depth overlays, and surface-normal maps per camera, so the devkit’s canonical visualization is reproducible directly in the App.
Fields.location, recording_date, run_number, plus has_rgb_lidar/has_depth/has_surface_normals/num_cameras capability flags. No object-level labels. The source dataset’s “ground truth” is millimeter-accurate TLS 3D models and centimeter-accurate trajectories, not per-sample annotations.
Good to know. Why is the Map tile empty for every episode? There’s no GNSS/GPS stream anywhere in the source dataset, so the App’s Map tile has nothing to show, by design, not a parsing gap.
Aquatic and off-road robotics datasets (raw, no object labels)
Same story as the ground-SLAM group: raw sensor streams plus a post-processed ground-truth pose, no object annotations. But the environments couldn’t be more different: a lake, a boreal forest, and a forest canopy from above.
CANOE
CANOE (Canadian Aquatic Navigation for Observation of the Environment) is ASRL’s marine navigation dataset, collected on an uncrewed surface vessel traversing a lake and reservoir in Ontario over three days. The sensor suite is genuinely unusual for a robotics dataset: a 360° Navtech scanning radar (dual range modes: 1000 m long-range or 150 m Doppler), a 128-beam Ouster LiDAR, a stereo camera, an Oculus imaging sonar, dual Starfish sidescan sonar, and a NovAtel GNSS/INS post-processed to 2-4 cm RMS accuracy. Four of the source’s 8 public sequences are included here, chosen for sensor/environment diversity: three fixed-duration windows plus the officially curated “Sample” sequence in full.
Fields.route, radar_max_range_m, source, plus has_image/has_pointcloud/has_gps capability flags. has_imu is False for every episode in this build even though an /imu topic exists, because it rides on a generic JSON channel rather than a schema-decoded one.
Keep it honest. Sidescan sonar is out of scope for this build. sidescan.xtf/sidescan.csv aggregate an entire sequence’s pings into single files with no per-frame breakdown, so they were never decoded into any MCAP channel.
FoMo is a year-long robotic data collection recorded in Forêt Montmorency, a boreal forest 80 km north of Quebec City, using a Clearpath Warthog UGV carrying two LiDARs (RoboSense + Leishen), an frequency-modulated continuous wave (FMCW) radar, a stereo + mono camera pair, two IMUs, stereo microphones, and PPK-GNSS ground truth. The full source is 9.43 TB across 60 sessions; this build samples 6 episodes, one per trajectory type, across 6 deployments spanning winter (-19°C, over a meter of snow) through summer conditions, kept under a 30 GB budget.
Keep it honest. This is a 6-of-60-session recon sample under a size budget, not a statistically representative subset. Don’t use it as a stand-in for the full dataset’s scale in a SLAM/odometry benchmark.
TreeScope is a LiDAR dataset for precision agriculture and forestry, collected across six forest and orchard sites with UAV and mobile-robot platforms. This build covers 10 episodes from just one site, VAT-0723 (Appomattox-Buckingham State Forest, Virginia), the only confirmed TreeScope site with released GPS data, making FiftyOne’s Map tile achievable at all. The UAV platform (a 4.2 kg Falcon 4) carries an Ouster OS1-64 LiDAR, stereo cameras, a VectorNav IMU, and a UBlox GPS. Each episode merges that session’s raw and Faster-LIO-processed ROS bag pair, remapping the processed trajectory to /refined/Odometry so both the onboard and offline-refined estimates stay distinguishable.
Fields.site, session_id, platform, lidar_model, flight-mode tags (manual_flight/autonomous_flight), plus has_image/has_pointcloud/has_gps/has_imu/has_logs flags.
Keep it honest. No ground truth is attached to any of these 10 episodes, deliberately. TreeScope’s site-level semantic-segmentation labels exist but their labeled-frame timestamps fall entirely outside every chosen episode’s flight window. metadata/VAT-0723.json independently confirms no semantic labels or diameter at breast height (DBH) measurements for these sessions. This is also a non-commercial dataset (CC BY-NC-SA 4.0).
TartanGround is CMU AirLab’s large-scale synthetic dataset for ground-robot perception and navigation, generated across a range of simulated environments. This build covers 6 trajectories, one per environment (AbandonedFactory, CyberPunkDowntown, GreatMarsh, Hospital, JapaneseCity, NordicHarbor), each converted to a native multimodal MCAP episode carrying a front camera, its paired segmentation stream, per-frame LiDAR point clouds, ego pose, and IMU telemetry on a shared 10 Hz frame clock.
Good to know. Unlike every other raw dataset in this post, TartanGround does ship a segmentation stream (/front-segmentation), but like the object-detection boxes elsewhere, it’s a synchronized MCAP channel decoded live in the Image tile, not a FiftyOne Segmentation label field.
This dataset card is shorter-sourced than the others in this post, verify field-level specifics directly before citing beyond what’s above.
These three are the exception to everything above: real, dense, per-frame object annotations, logged as additional timestamped channels inside the same MCAP timeline as the sensor data, decoded live by the multimodal viewer’s Image and 3D tiles in sync with playback, rather than stored as separate FiftyOne label fields.
SemanticSpray++
SemanticSpray++ labels a 36-scene subset of the RoadSpray dataset: an ego vehicle following a lead vehicle (a small car or a large van) on wet pavement at matched speeds from 50-130 km/h, generating a trailing water-spray plume off the wet surface. Every one of the 36 episodes in this build has the complete label set: 2D camera boxes, 3D LiDAR boxes, LiDAR point-wise semantics (background/foreground/noise), and radar point-wise semantics, all for the lead vehicle. Across the 36 episodes: 1,362 Car and 1,225 Van 2D/3D box instances, plus 77.8 million labeled LiDAR points.
Keep it honest. The other 171 scenes in the same source download (the earlier, LiDAR-only “SemanticSpray” release with no boxes) are deliberately not included. This build targets exactly the 36-scene, fully-labeled SemanticSpray++ subset.
Boreas is ASRL’s multi-season driving dataset, collected by repeatedly driving a route near the University of Toronto over a full year: sun, rain, falling snow. Boreas Road Trip (Boreas-RT) extends it with 9 new, more diverse routes and adds an Aeva FMCW Doppler LiDAR plus a standalone IMU. A separate dedicated recording, boreas-objects-v1, provides 326,180 3D object annotations (Car/Pedestrian/Cyclist/Misc) across 7,111 labeled LiDAR frames. This build covers 2 driving episodes (one Boreas, one Boreas-RT, each a 10-second window) plus 6 object-detection windows carved from boreas-objects-v1.
Fields.episode_type (odometry/object_detection), dataset_variant, has_boxes, has_aeva, has_dmu_imu. Label channels (object-detection episodes only): /boxes (3D SceneUpdate cuboids), /camera/boxes and /radar/boxes (2D wireframe overlays), all carrying label/uuid/numPoints metadata inspectable via FiftyOne’s click-to-inspect panel.
Keep it honest.has_boxes is True for only 6 of the 8 episodes. The 2 driving/odometry episodes carry no object annotations at all. Only obj_train-split windows are included; obj_test ground truth is withheld by the source dataset for leaderboard fairness.
The CMHT Autonomous Dataset is McMaster University’s multi-sensor driving dataset recorded around Hamilton, Ontario: a single rig combining a Velodyne LiDAR, an mmWave radar, a monocular RGB camera, and a FLIR thermal/IR camera, across four drives spanning dusk/clear and night/rain conditions. Over 9,000 frames are labeled with 3D position, size, rotation, class (Car/Truck/Van/Pedestrian/Bus/LongVehicle), and a persistent object ID. This repackaging re-embeds those separately-published labels back into each drive’s MCAP timeline, synced to the exact sensor message each was originally annotated from.
Fields.drive_name, weather, lighting, is_preliminary_sync, has_3d_labels, has_2d_labels. Label channels: /Labels_3D (vision_msgs/Detection3DArray, one message per labeled LiDAR frame) and /Labels_2D_front//Labels_2D_ir (vision_msgs/Detection2DArray).
Keep it honest. The 2D boxes are derived, not manually annotated. Only the 3D LiDAR-frame boxes were hand-labeled (via SUSTechPoints); the 2D camera and thermal boxes are projections of those same 3D boxes onto each image plane, computed by this repackaging using the authors’ own calibration and projection method. is_preliminary_sync is also True for 3 of the 4 drives: only the LiDAR triggers the cameras, while radar, GPS, and IMU are approximately, not frame-exactly, time-aligned.
NavWareSet is a multimodal dataset of socially-compliant and non-compliant robot navigation around human pedestrians, recorded in a controlled indoor environment across seven canonical social-navigation scenarios (Frontal Approach, Pedestrian Obstruction, Blind Corner, Following Human, Perpendicular Traffic, Circular Crossing, Object Handover). Every scene is captured simultaneously by the robot’s own sensors and by a stationary external Ground-truth Recording Station (a LiDAR mounted above a camera), which also provides the point cloud human trajectories were manually annotated on. Seven of the source’s 48 scenes are included here, Clearpath Jackal platform only. The Toyota HSR platform isn’t imported yet, though the merge pipeline already supports it.
Fields.scenario, robot_platform, behavior (social/non-social), participant_group, has_pedestrian_annotations. Label channel: /annotations/pedestrians (foxglove.SceneUpdate, persistent per-person cuboids carrying the source’s own tracking ID).
Good to know. Why isn’t there a Detections field for the pedestrian boxes? Pedestrian cuboids are a time-synced channel meant to play back alongside the LiDAR/camera channels in the 3D tile: SceneUpdate entities carry a persistent entity ID across frames, matching the source annotation format’s per-person UUID directly, with no separate tracking field needed.
Across all 10 datasets, the actual training bottleneck is real, timestamped, multi-sensor recordings of something moving through the world (a car, a boat, a UAV, a robot navigating around a person) with enough synchronized modalities to reconstruct what happened. That’s the raw material world models and embodied agents train on, and it’s the shape of data MCAP was built to hold and FiftyOne’s multimodal viewer was built to make explorable without a custom bag-parsing script.
MCAP is a modular, serialization-agnostic binary container format for timestamped pub/sub data: one file holds any number of named channels (camera, LiDAR, IMU, GPS, transforms), each indexed for fast seeking. It’s been the default log format for ROS 2 since the Iron release.
Not directly. ROS 1 .bag and ROS 2 .db3 files need to be converted to .mcap first (via the mcap CLI, rosbags, or a custom script), at which point FiftyOne’s media_type: "multimodal" support decodes them automatically based on the .mcap file extension.
Only three: semanticspray-plusplus (all 36 episodes, 2D boxes, 3D boxes, point-wise semantics), boreas-multimodal (6 of its 8 episodes, via the has_boxes field), and cmht-autonomous-driving (3D boxes plus derived 2D projections, all 4 episodes). navwareset has real pedestrian cuboids but as a SceneUpdate channel, not a FiftyOne label field. The remaining 6 are raw sensor data with no object annotations at all.
A multimodal dataset (what every dataset in this post is) has media_type: "multimodal" and one sample per .mcap episode, with every stream inside decoded live by the viewer’s tiles. A grouped dataset instead has media_type: "group" and multiple related samples (e.g., different camera slices, a 3D scene) linked under one group field, a different mechanism used when labels need to attach as ordinary per-frame FiftyOne fields rather than live inside an MCAP timeline.
Check each license individually. semanticspray-plusplus, fomo-multimodal-sample, canoe-multimodal, navwareset, and TartanGround are CC BY 4.0; cmht-autonomous-driving is CC0 1.0. oxford-spires-multimodal and treescope-vat0723-multimodal are CC BY-NC-SA 4.0 (non-commercial only). boreas-multimodal’s license is unlisted. yuto-mms-multimodal has a genuine conflict between its project site (CC BY-NC-SA) and its live Zenodo records (CC BY). Verify directly before relying on either.
