FiftyOne 1.19 can now open MCAP files natively — camera, LiDAR, GPS, IMU, and logs, all on one shared clock, no ROS install required.
A UAV flies under the canopy of a Virginia forest for four and a half minutes. In that window, it isn't recording one thing — it's recording thirty-one things at once. Stereo cameras. A LiDAR sweeping 64 channels at 1024 points each. GPS fixes. An IMU. Onboard diagnostics. Two competing estimates of its own position, one live, one refined after the fact. Every one of those streams ticks on its own clock, at its own rate, into a single append-only log file.
That file is real: it's VAT-0723U-AUTO-01_episode.mcap, one sample in a dataset we built for this post, and by the time the flight is over, it holds 599,698 messages across 31 channels. Nothing about that file tells you what a "frame" is. There's no sample, no image, no obvious unit at all — just a firehose of timestamped messages on parallel channels, exactly as the robot's middleware wrote them.
Historically, opening that file meant leaving your dataset tool behind. You reached for rosbag, or Foxglove, or rviz — because whatever held your curated samples had no idea what to do with a heterogeneous channel firehose. As of FiftyOne 1.19, that's no longer true. Point a Sample at a .mcap file, and FiftyOne opens it: cameras,
point clouds, GPS tracks, IMU plots, and logs, all synced to one playback clock, inside the same App you already use for images and video.
Key Takeaways
- MCAP is a self-describing log container, not a dataset format — every message's schema is encoded in the file, so a recording remains decodable years after the code that wrote it is gone.
- Point a FiftyOne
Sample's filepath at a .mcap file, and dataset.media_type is automatically inferred as "multimodal" — no importer class, no config, no media-type flag. - The App's multimodal viewer renders six tile types — Image, 3D, Map, Plot, Logs, and Message — all synced to one shared playback clock, so scrubbing the timeline moves every tile at once.
- Built-in decoders support both ROS 1 and ROS 2 message encodings, plus Foxglove's protobuf/CDR schemas, with no ROS installation required on your machine.
- In the 10-episode UAV LiDAR dataset built for this post, one autonomous flight (VAT-0723U-AUTO-01, ~4.5 minutes) alone carries 31 channels and nearly 600K messages across stereo cameras, LiDAR, IMU, GPS, and logs — all playable from a single sample.
- Anything FiftyOne doesn't have a built-in decoder for — raw Ouster LiDAR packets, custom drone-autonomy messages — still opens in the Message tile as a raw, collapsible record tree, so nothing in the file is ever invisible, even unrecognized data.
What is MCAP, actually?
Strip away the robotics branding, and MCAP is a log container, not a dataset format. It's the modern successor to ROS 1's .bag and the SQLite-backed rosbag2, and it's been the default output of ros2 bag record since the ROS 2 Iron release. Its entire reason for existing is recording timestamped publish/subscribe message streams to disk, fast and durably, in a format that survives a crash mid-write.
Four record types carry the whole mental model:
- Schema — a serialization-agnostic message definition (ROS msg, protobuf, JSON schema, FlatBuffers). This is what makes the file self-describing.
- Channel — a named stream, essentially an ROS topic. Each channel maps to a single schema and specifies its message encoding.
- Message — one record: a channel ID, a sequence number, a log time, a publish time, and an opaque serialized payload.
- Chunk (plus index records) — messages are batched into compressed chunks with a summary section at the end, so a reader can seek by time or by channel without scanning the whole file.
The hierarchy is File → Channels → time-ordered Messages → decoded via Schemas. The primary axis is time, and the data on that axis is heterogeneous by design — camera frames on one channel, LiDAR sweeps on another, IMU, GPS and diagnostics on others, all interleaved by timestamp with no shared "frame" concept tying them together.
Good to know. Why does the schema live inside the file instead of in code?
Because then, months or years later, any tool without access to the original ROS packages can still decode every message. That's exactly why a raw ouster_ros/msg/PacketMsg blob and a self-describing sensor_msgs/msg/PointCloud2 message are both fully readable at the byte level in our dataset — even though, as you'll see below, only one of them means anything to a generic viewer without vendor-specific decode logic.
Why your dataset tool and your robot's memory used to be two different tools
A FiftyOne dataset's unit is the sample — a piece of media with fields and labels. An MCAP file has no notion of a sample at all. It has channels, ticking independently, and the question "what even is one data point here?" has no answer baked into the format. Turning that channel firehose into curated training samples — deciding which channels sync into which moment, and what time tolerance counts as "the same moment" — is a real, separate data-curation problem, and it still is one.
What changed in FiftyOne 1.19 is narrower and more immediately useful: you no longer need a second tool just to look at what's in the file. The multimodal viewer reads the MCAP directly via byte-range reads — no server-side conversion, no pre-processing step — discovers its channels and schemas, and renders whatever it can decode in a tiled, scrubbable viewer.
Keep it honest. Deciding which recorded moments become labeled training samples is still a modeling decision you have to make yourself; FiftyOne's multimodal viewer doesn't make it for you. What it gives you is the ability to see everything a recording contains across every channel before you make that call — without spinning up Foxglove or rviz alongside your dataset tool.
Meet the dataset: 10 UAV flights through a Virginia forest
TreeScope is a
LiDAR dataset for precision agriculture and forestry, collected by the GRASP Laboratory at the University of Pennsylvania with UAV and mobile-robot platforms across six forest and orchard sites. Its original release ships raw and processed ROS 1 bags, HDF5 semantic labels, and field-measured tree diameters — built for benchmarking segmentation and diameter-estimation algorithms, not for showing off a multimodal viewer.
For this post, we built a 10-episode subset from one TreeScope site, VAT-0723 (Appomattox-Buckingham State Forest, Virginia), where each sample is one continuous UAV flight, merged from that flight's raw and processed ROS 1 bag pair into a single .mcap file. VAT-0723 was picked deliberately: it's the one TreeScope site we could confirm still has real GPS (NavSatFix) data in its released bags, which is what makes the Map tile demonstrable at all — the TreeScope paper mentions GPS hardware broadly, but it isn't present in every site's public release.
Four episodes are manually piloted flights; six are autonomous. Durations range from 36 seconds to 6.4 minutes. Nine of the ten have every tile type populated with real, decodable data — the tenth (VAT-0723U-01) genuinely lacks camera and IMU topics, and we kept it in rather than drop it, because a dataset where every sample has every sensor isn't realistic.
Keep it honest. This is a 10-episode slice from one site, not the ~2.2 TB, six-site TreeScope corpus, and it does not include TreeScope's own semantic segmentation or diameter ground truth — the timestamps for those labels don't overlap with the specific episodes we chose. What's attached instead is schema-derived metadata (has_image, has_gps, and so on) that describes what each episode's MCAP actually contains, not annotations.
Step 1: Point FiftyOne at your .mcap files
That's the entire import. No dataset-type argument, no schema mapping. FiftyOne infers media_type: "multimodal" straight from the .mcap extension, the same way it infers image from a .jpg.
Step 2: Let FiftyOne tell you what's actually decodable — don't guess from topic names
A recording's topic list is tempting to pattern-match on, and it will burn you. While building this dataset, an early version tagged VAT-0723U-01 as having IMU data because its topic list includes /os_node/imu_packets — but that topic is ouster_ros/msg/PacketMsg, a raw undecoded Ouster UDP blob, not a real sensor_msgs/msg/Imu stream. It will never render in the Plot tile, no matter what its topic name suggests.
The fix is checking schema names, not topic names, and storing the result as ordinary sample fields:
Every episode in this dataset carries has_image, has_pointcloud, has_gps, has_imu, has_logs, and has_unrecognized_schema fields built this way — which is also what makes it possible to query the dataset for exactly the episodes worth demoing (more on that below).
Step 3: Open the tiled viewer and wire up tiles
Open any multimodal sample, and the App gives you a mosaic-style viewer built from tiles — Image, 3D, Map, Plot, Logs, and Message — that you add, remove, resize, and rearrange. The Add tile button (grid icon at the top of the viewer) lets you pick a type; the left sidebar's Topics tab lists every channel in the recording, grouped by category, so you can see at a glance which tiles a given topic can feed into.
Bind each tile to a stream, and every tile shares one playback clock. Scrub the timeline once, and the Image tile's camera frame, the 3D tile's point cloud, and the Map tile's GPS marker all advance together — because underneath, they're all being read from the same set of timestamped messages in the same file.
Step 4: Scrub the timeline and watch it stay honest
On VAT-0723U-AUTO-01, scrubbing forward moves through a real ~4.5-minute autonomous flight: the /ublox/fix GPS track moves across a small span of the Appomattox-Buckingham forest (latitude 37.45492–37.45507, longitude −78.61621 to −78.61609, altitude 173–186 m), the stereo /ovc/left and /ovc/right cameras update frame by frame, and the /tree_cloud channel plays back 453 organized 64×1024 point-cloud sweeps — the same resolution as the Ouster OS1-64 that captured them.
Add a Plot tile on /ovc/vectornav/imu's linear_acceleration, and the y-axis sits at a median of about −9.3 m/s² for most of the flight — gravity, projected onto that axis given how the IMU is mounted — then swings sharply during real flight maneuvers. Bind a second Plot tile to both /Odometry and /refined/Odometry's pose.pose.position.z, and you're watching two independent height estimates of the same flight: the live onboard estimate and the offline, Faster-LIO-refined one agree to within about 3 cm most of the time (median absolute difference), but momentarily pull apart by as much as 0.84 m at points where the offline correction has more to fix.
Step 5: Fall back to the Message tile for anything else
Every episode in this dataset also includes channels that FiftyOne has no built-in decoders for: raw ouster_ros/msg/PacketMsg LiDAR packets (undecoded UDP payloads straight off the sensor) and, on the autonomous flights, custom quadrotor-autonomy schemas like kr_mav_msgs/msg/PositionCommand and planning_ros_msgs/msg/VoxelMap. None of that data is hidden. Bind a Message tile to any of those topics, and you get the most recent message at the current playhead as a collapsible record tree — every field, still inspectable, even though nothing about the schema was known to FiftyOne ahead of time.
Nine things to try in the multimodal viewer
Grounded in the actual dataset above — here's what we'd record as GIFs to show this off, in the order they'd appear in a post or demo.
- The grid, at a glance. Load all 10 episodes, let the grid render its point cloud and camera previews, then use the stream selector to switch which channel each card previews. Follow with applying the
full_multimodal_showcase saved view to filter down to the 9 episodes with full tile coverage — a clean before/after of curation by capability, not by eyeballing files. - Four tiles, one clock. Open
VAT-0723U-AUTO-01, arrange Image ×2 (/ovc/left, /ovc/right), 3D (/tree_cloud or /os_node/segmented_point_cloud_no_destagger), and Map (/ublox/fix) tiles, then hit play. This is the single best "one file, every sensor, one clock" shot — cameras, point cloud, and GPS marker all advancing together over the ~4.5-minute flight. - IMU under a maneuver. Add a Plot tile on
/ovc/vectornav/imu's linear_acceleration (x, y, z) on VAT-0723U-AUTO-01 and scrub to a section where the value swings well past its ~−9.3 m/s² gravity baseline — a visible flight maneuver on a live chart, with a playhead marker tracking the scrub position. - Onboard vs. refined trajectory. A second Plot tile comparing
/Odometry and /refined/Odometry on pose.pose.position.z — the live onboard estimate against the offline Faster-LIO-refined one, agreeing to within a few centimeters most of the time but visibly pulling apart (by up to 0.84 m) at specific points in the flight. - The Message tile as an escape hatch. Bind a Message tile to
/quadrotor/position_cmd (kr_mav_msgs/msg/PositionCommand) or raw /os_node/lidar_packets (ouster_ros/msg/PacketMsg) and scrub the timeline, showing the raw record tree update at the playhead — proof that even fully custom, undecoded schemas stay inspectable. - Logs scrolling in sync. Add a Logs tile on
/rosout_agg on any episode and let it scroll during playback, then pause following to scan backward through the history — a console view moving in lockstep with the same clock as the 3D and Map tiles. - A sparse episode, honestly shown. Open
VAT-0723U-01 — the one episode with no camera or IMU — and show the Topics sidebar listing what's actually available, followed by a Message tile on one of its unrecognized-schema topics. Pairs well with clip #1's "full coverage" view to show FiftyOne handling partial sensor coverage gracefully rather than only ever showing the best case. - Live temporal tagging. Shift-click-and-drag along the timeline on any episode to create a temporal tag on the spot (e.g. tagging a GPS-track segment or a camera-visible landmark) — this dataset has no pre-existing temporal tags (none of its ground truth overlaps these specific flights), so this should be captured as a live "here's how you'd mark an event of interest" action, not as a reveal of existing annotations.
- The MCAP Explorer, no dataset required. Drag a standalone
.mcap file straight into the MCAP Explorer panel and show it rendering immediately — useful as a closing beat to demonstrate you don't even need a FiftyOne dataset to start looking at a recording.
Frequently Asked Questions
Do I need ROS installed to view MCAP files in FiftyOne?
No. MCAP embeds each channel's schema directly in the file, so FiftyOne's built-in decoders can read ROS 1, ROS 2, Foxglove (protobuf and CDR), and JSON-encoded messages without a ROS installation on your machine.
What happens to channels FiftyOne doesn't know how to decode?
They're never hidden. Any channel — decoded or not — is accessible in the Message tile as a raw, collapsible record tree at the current playhead, which is exactly how this dataset's raw Ouster LiDAR packets and custom quadrotor-autonomy messages stay inspectable.
Can I query which episodes have which sensors, instead of opening each one?
Yes — that's what schema-derived fields like has_image and has_gps are for. Build them once at import time (see Step 2), and they work like any other FiftyOne field: dataset.match(F("has_image") & F("has_gps")), or a saved view like full_multimodal_showcase in this dataset.
Is this the same as FiftyOne's MCAP indexing / event mining?
No. What's covered here — the tiled viewer, playback, temporal tags — is available in open-source FiftyOne. Indexing MCAP data into queryable Parquet tables for fleet-scale event mining (finding every episode where a condition held, across thousands of recordings) is a FiftyOne Enterprise feature and isn't used anywhere in this dataset.
Does FiftyOne convert the MCAP file to something else on import?
No. The multimodal viewer reads the .mcap file directly via byte-range reads when you open a sample — there's no server-side conversion step and no intermediate format.
A robot's memory used to require its own tooling to even look at. Now it opens next to your images and your videos, on the same clock, in the same App.
