Download one Aria Digital Twin episode, author every ground-truth stream it ships with into a single synchronized MCAP file, and load it into FiftyOne’s multimodal viewer, using Meta’s own capture pipeline and about 300 lines of Python.
Aria Digital Twin (ADT) is a research dataset from Meta Reality Labs: real people performing everyday activities in two fully digitized rooms, recorded with Project Aria smart glasses and paired with millimeter-accurate six degrees of freedom (6DoF) ground truth for every object, person, and device in the scene. It ships as a .vrs recording plus eight separate ground-truth zip files, each behind its own presigned URL. None of that is something FiftyOne (an open-source toolkit for visualizing and curating ML datasets; pip install fiftyone) opens directly.
What FiftyOne’s multimodal viewer wants is one file per episode: a single timeline where the RGB video, depth, segmentation, synthetic renders, 2D and 3D object boxes, human skeletons, eye gaze, the semi-dense point cloud, the device’s 6DoF trajectory, and the inertial measurement unit (IMU) stream all play back in sync. That file is an MCAP: a container format for timestamped, multi-channel binary data, the same idea as a ROS bag but not tied to ROS.
Here’s the whole path from Meta’s download page to a dataset you can scrub through in FiftyOne, for one Apartment-scene episode, with every modality ADT ships turned into its own MCAP channel.
Keep it honest. Aria Digital Twin (Meta Reality Labs Research, arXiv:2306.06362) is licensed for personal, non-commercial research only, and the license explicitly bars distributing or publicly displaying the dataset “or any materials or works derived from it.” You need to accept that license yourself through Meta’s Aria Dataset Explorer before you get download links. This post shows code, not a copy of the data.
Key takeaways
Aria Digital Twin (236 sequences as of this writing, up from the 200 in the original paper) ships as a raw .vrs recording plus ground-truth zips, served from presigned URLs gated by Meta’s license, six of which this pipeline actually reads.
One episode with every modality is a real download: 12.4 GB across seven files for a 145-second sequence, with the raw depth zip alone accounting for 7.8 GB of that.
AriaDigitalTwinDataProvider hands you per-frame images, boxes, skeletons, and gaze by timestamp; twelve foxglove channels turn those into one .mcap file FiftyOne plays natively, with no exporter or custom loader.
Two silent gotchas: ADT’s world frame is Y-up while Foxglove/ROS 3D viewers assume Z-up, and writing each modality in its own loop desyncs playback unless you re-sort every message by log_time afterward.
A third gotcha only 3D boxes hit: 304 of this episode’s 354 tracked objects never move. Redrawing all of them every frame made that one channel 77 MB; logging the static ones once and only redrawing the 50 that actually move is the fix.
The payoff: a 145-second episode becomes one 2.5 GB .mcap with 181,444 messages across 12 channels, synchronized, one fo.Sample(filepath=...) call away from FiftyOne’s multimodal viewer.
Every code block in this post ran against foxglove-sdk 0.26.0, projectaria-tools 2.2.0, mcap 1.3.1, pillow 11.3.0, numpy 2.4.6, pandas 3.0.0, and fiftyone 1.21.0. You’ll also need curl on your PATH for Step 1’s downloads.
Step 1: Download one Aria Digital Twin episode
Request access to ADT through the Aria Dataset Explorer or the dataset download page, and accept the license. Meta returns a single JSON file, commonly called a “CDN file,” with two top-level keys: sequences (every sequence name, each mapping component name to a presigned download URL, file size, and checksum) and sequence_config (which components exist). Save it locally; this post calls it adt_download_urls.json, but yours may be named differently. manifest["sequences"].keys() lists every sequence name available to pick from, including Apartment_release_multiskeleton_party_seq101_M1292, the one used throughout this post.
Meta ships an official downloader for this file: pip install projectaria-tools (already in this post’s Installation step) installs an aria_dataset_downloader command that reads a CDN file directly:
That downloads the raw files. Every modality below needs its own component from the CDN file:
The seven ADT components this pipeline downloads, what each one backs in the final MCAP, and its size for Apartment_release_multiskeleton_party_seq101_M1292
The seven ADT components this pipeline downloads, what each one backs in the final MCAP, and its size for Apartment_release_multiskeleton_party_seq101_M1292
Component
Backs
Size (this episode)
main_vrs
RGB/SLAM images, IMU, calibration
2,341 MB
main_groundtruth
2D/3D boxes, instances, skeleton files, raw eye gaze
97 MB
mps_slam_trajectories
Device 6DoF pose
15 MB
mps_slam_points
Semi-dense point cloud
624 MB
segmentation
Instance segmentation video
573 MB
depth
Per-pixel metric depth video
7,792 MB
synthetic
Photo-realistic synthetic render
918 MB
To see exactly what’s happening, and to go straight into the flat, extracted layout this pipeline expects, here’s the equivalent in plain Python instead of the CLI:
Good to know. Why not mps_slam_calibration or mps_eye_gaze? They’re on Meta’s own recommended download list, but nothing in this pipeline reads online_calibration.jsonl or the MPS copy of general_eye_gaze.csv: eye gaze comes from eyegaze.csv in main_groundtruth, and camera calibration comes from the VRS file’s own embedded calibration, not the MPS refinement pass. AriaDigitalTwinDataPathsProvider.get_datapaths() doesn’t reference either file, which is how to check this for any zip you’re tempted to skip.
AriaDigitalTwinDataProvider expects a specific directory layout: video.vrs plus an mps/slam/ subfolder. Symlink everything into that layout:
Step 2: Author the MCAP
This continues in the same script as Step 1: seq_dir is the directory you just built there. Everything below writes into one foxglove.open_mcap(...) block. Set up the providers, instance metadata, and camera geometry first, since several streams share them:
Good to know. Doesn’t MCAP need real calendar timestamps for log_time? No. log_time just needs to be in nanoseconds on a single consistent clock; MCAP doesn’t require wall-clock Unix epoch time. ts_rgb is nanoseconds since the Aria device booted, which puts this episode’s timestamps around 30 minutes past epoch. Any tool that prints log_time as a calendar date shows something in January 1970. That’s expected for this dataset, not a unit bug, and it doesn’t affect playback or sync, since every stream below shares the same clock.
Good to know. Where does the axis and rotation math live? Three small helpers do all the geometry correction, reused by every 3D and image stream below:
The 3D boxes: split static from dynamic before you write a single frame
ADT tags every object instance’s motion_type. In this episode, 304 of 354 tracked objects (furniture, wall art, kitchenware) never move at all; only 50 (people, handled objects) do.
Keep it honest. Why does this matter enough to be step one? Redrawing all 354 boxes as a full wipe-and-rebuild every frame at 30 Hz and /scene/3d_boxes alone comes out to around 600 MB, roughly the same size as the RGB stream, because most of that geometry never changes. Logging the 304 static ones once with a persistent lifetime and only redrawing the 50 that move dropped this episode’s dynamic-box channel to 77 MB. The 3D tile has to rebuild far less geometry every frame, which is what keeps it from lagging behind the video.
A point cloud logged at a single timestamp disappears from the 3D tile once playback scrubs past it. Republish the same encoded bytes roughly once a second instead:
Good to know. Why does the downsample rate matter here? A FrameTransform that carries a timestamp is a dynamic transform, and dynamic transforms only resolve within a boundary clamp of the current playhead (50 ms by default in FiftyOne’s viewer). At ~50 Hz, updates land every 20 ms, well inside that window. Downsample much further, to 10 Hz say, and the device marker starts flickering or briefly vanishing between updates instead of gliding smoothly.
The per-frame loop: RGB, gaze, boxes, skeleton, depth, segmentation, synthetic
Everything else is keyed to the RGB camera’s own timestamps, one iteration per frame:
Keep it honest. Why rotate the image, and rotate the boxes differently? Aria’s cameras are physically mounted rotated, so every image stream needs the same rotate_cw90. A 90° clockwise pixel rotation maps (x, y) to (height - 1 - y, x), which is why the 2D box coordinates above get remapped with img_h, not just copied over. Skip either one and the boxes end up floating next to the objects they’re supposed to outline instead of on them.
IMU comes straight off the raw VRS provider, not through the ground-truth API:
That’s episode_raw.mcap done, with 12 channels written in 5 separate passes (static boxes, point cloud, pose, the big per-frame loop, IMU). One step left, and skipping it is expensive:
Keep it honest. Why reorder at all? Each of those 5 passes writes its channel’s messages as one contiguous block spanning the full 145-second episode. A reader stepping through time then has to jump between physically distant regions of the file, once per channel, on every playback frame. That’s what a stuttering, out-of-sync multimodal viewer usually means. reorder_mcap re-reads every message in log_time order and rewrites them so that each disk chunk holds a single narrow time slice across all channels instead.
Run it end to end, and you get real numbers: 4,349 RGB frames, matching depth/segmentation/synthetic frames, 2,832 annotation and skeleton messages, 150 point cloud republishes, 7,198 pose messages, 145,371 IMU samples, and one static-box message covering 304 objects, for 181,444 messages total in one 2.5 GB file.
Good to know. Is a clean run enough to trust the file? No. Message counts and a successful reorder both pass even on a file with a real geometry bug baked in, the same way a program can run to completion and still return the wrong answer. reorder_mcap does run a file-integrity check internally (which catches truncation and sparse-file holes), but that’s a structural check, not a correctness check. Add at least one physical-plausibility assertion before you trust the output. For the point cloud above, this episode’s semi-dense points span about 17-21 meters per axis, apartment-scale, which is the kind of check that catches a stray unit conversion or a botched axis swap that a message count never would:
Step 3: Create a FiftyOne dataset
This is the short part. FiftyOne treats one .mcap file as one multimodal sample:
That’s it. FiftyOne reads the file directly, so there’s no exporter or custom loader to write. Open the sample in the App and add tiles for whichever channels you want: RGB, depth, segmentation, and synthetic as Image tiles; 3D boxes/skeleton/gaze/point cloud/trajectory as 3D tiles; and IMU as a Plot tile, all sharing one playback clock. For more than one episode, loop add_sample() over a directory of .mcap files and attach whatever per-episode fields you want to filter on later, such as duration, scene name, or which streams are present.
Frequently Asked Questions
Because FiftyOne’s multimodal viewer decodes the same well-known foxglove.* protobuf schemas that Foxglove Studio does. Authoring against foxglove-sdk produces a file that opens in either tool unmodified; nothing here is Foxglove-Studio-specific.
Only if you want every stream. main_vrs plus mps_slam_trajectories alone gets you RGB, pose, and IMU. Each additional zip (main_groundtruth, segmentation, depth, synthetic, mps_slam_points) unlocks exactly the streams named in the table in Step 1, nothing else.
No. Meta’s license prohibits distributing or publicly displaying the dataset “or any materials or works derived from it,” and that includes an MCAP you author from it. Keep it local, or in a private repo you control.
It ships as raw, uncompressed per-pixel metric depth video, 7.8 GB for one 145-second episode before this pipeline ever touches it. The PNG-encoded /camera/depth channel in the final MCAP is a small fraction of that, since PNG compresses the mostly-smooth depth values well.
Two usual suspects, both covered above: you skipped the reorder_mcap pass, or you’re redrawing every 3D object, including the 304 out of 354 that never move, on every single frame instead of splitting static from dynamic.
