How to turn a 56GB drone photogrammetry dataset into MCAP episodes FiftyOne can scrub through in 3D, using hf download, foxglove-sdk, and one coordinate-conversion function.
PIVOT ships as five folders of JPEGs and one big scene_data.json per scene. Open the dataset card, and it reads like every other NeRF/3D Gaussian splatting benchmark: scenes, trajectories, camera poses, a sparse point cloud. Nothing about that layout says “robotics dataset.”
But look at what’s actually inside scene_data.json. Every frame carries two camera poses, not one: a measured pose from the drone’s GPS, flight attitude, and gimbal angle, and a COLMAP-optimized pose computed offline from the images themselves. Every trajectory shares one static sparse point cloud with every other trajectory in its scene. And every trajectory is, structurally, a single camera moving through a fixed 3D map over time.
PIVOT reads like a NeRF benchmark, but its trajectory structure is a robot log.
MCAP is built for exactly that: timestamped messages on named topics, each with a declared schema, all replayable in sync. Two poses per frame become two /tf transforms against one map. One point cloud becomes one message logged at the start of the episode, rather than duplicated per frame. This post covers the three steps to get from PIVOT’s raw files to a FiftyOne dataset you can scrub through: download only what you need, author one MCAP per trajectory, load the episodesas samples.
Key takeaways
PIVOT advertises 56GB, but 26GB of that is PYCOLMAP_soft_prior/ — COLMAP’s disposable feature-matching database and candidate reconstructions, already summarized into scene_data.json and sparse_model.ply. Excluding it with hf download --exclude cuts the real download to about 27GB.
Each trajectory becomes one .mcap file with four topics: /camera/image_raw (foxglove.CompressedImage), /camera/calibration (foxglove.CameraCalibration), /tf (foxglove.FrameTransform, logged per frame for each pose source), and /map/points (foxglove.PointCloud, logged once).
PIVOT’s world frame is NED (north, east, down) with an OpenGL-style camera convention; Foxglove and FiftyOne expect a Z-up world and an OpenCV/ROS optical camera. Two 3x3 rotation matrices, applied consistently to poses and point-cloud vertices, fix both.
COLMAP doesn’t register every frame. church/rocket_upward has zero registered frames. The code below checks for colmap_pose_c2w per frame and skips the transform if it’s missing, rather than fabricating a pose.
The resulting FiftyOne dataset has media_type == "multimodal": one sample per trajectory, and opening a sample plays the camera moving through the scene’s point cloud.
huggingface_hub[cli] provides the hf command used in Step 1. foxglove-sdk provides the foxglove package used to author MCAP files in Step 2, and plyfile reads PIVOT’s .ply point clouds. fiftyone loads the finished episodes in Step 3.
Budget disk space for both copies of the data at once: about 27GB for the downloaded JPEGs/JSON/PLY files from Step 1, plus another ~28GB for the .mcap episodes Step 2 writes alongside them, so roughly 55GB free before you start.
Step 1: Download only the data you need
PIVOT’s Hugging Face repo is 56GB, but half of that is COLMAP’s intermediate working files: a multi-gigabyte feature-matching database and several candidate sparse reconstructions per scene, all superseded by the curated scene_data.json and sparse_model.ply that PIVOT already ships alongside them. Exclude that directory and the download drops to about 27GB:
That leaves this layout on disk, one folder per scene under scenes/:
Five scenes, 103 trajectories in total. Step 2 reads scene_data.json and sparse_model.ply directly; it never touches the JPEGs except through the file names listed inside scene_data.json. A single frame entry inside that file looks like this:
colmap_pose_c2w is present on registered frames. On frames COLMAP couldn’t place, the key is absent entirely, not set to null.
Step 2: Author one MCAP per trajectory
PIVOT’s poses are 4x4 camera-to-world matrices in the NED world frame (X-north, Y-east, Z-down) with an OpenGL-style camera (X-right, Y-up, Z-backward). Foxglove expects a Z-up world and an OpenCV/ROS optical camera (X right, Y down, Z forward). Two rotation matrices, applied the same way to every pose and to the point cloud, handle both:
PIVOT ships no real per-frame capture timestamps — the JPEG EXIF timestamps are stripped; only GPS and gimbal angles survive. The MCAP uses a synthetic 10 fps sequence clock instead, so frame order is preserved without claiming a real elapsed flight time:
Good to know. Why does calibration render silently fail so often?
Pairing an image with a CameraCalibration message has four independent gates, and missing any one of them just gives you an image with no frustum, no error: the calibration message has to exist, the image channel has to name its calibration topic in metadata (as above), the image topic’s last path segment has to contain image plus raw/rect/rectified, and distortion_model has to be spelled exactly plumb_bob, rational_polynomial, equidistant, or fisheye — kannala_brandt, for instance, is rejected even though it’s the same model as equidistant.
Good to know. Why two /tf transforms per frame instead of one?
PIVOT’s whole design rests on comparing pose sources: measured_pose_c2w comes from the drone’s onboard sensors, colmap_pose_c2w comes from offline structure-from-motion. Logging both against the same /map/points on separate child frames (camera and camera_colmap) keeps that comparison intact rather than collapsing it to a single “correct” pose.
Keep it honest.
This trajectory (orbit_inward_low) uses a standard-field of view (FOV) lens, so a 5-coefficient plumb_bob model is enough. PIVOT’s wide-FOV trajectories use a genuine 4-coefficient fisheye fit, and at least one of those fits turns non-invertible near the edge of the frame. That’s a real gotcha, but a distortion-model detail rather than part of the MCAP-authoring story, so it’s left out here.
The five message streams in one PIVOT episode, and how often each one is logged.
The five message streams in one PIVOT episode, and how often each one is logged.
Topic
Foxglove schema
Logged
What it carries
/camera/image_raw
CompressedImage
Every frame
The original JPEG bytes, unmodified
/camera/calibration
CameraCalibration
Every frame
Trajectory intrinsics and plumb_bob distortion, repeated so each timestamp is self-describing
/tf (world to camera)
FrameTransform
Every frame
The measured pose from the drone's GPS, flight attitude, and gimbal angle
/tf (world to camera_colmap)
FrameTransform
Registered frames only
The COLMAP-optimized pose, skipped when the key is absent
/map/points
PointCloud
Once, at log_time=0
The scene's static sparse point cloud, shared by every trajectory
Validate before you batch.
A message-count check is not proof the file is good — one MCAP conversion can report every channel count matching its source and still contain a multi-gigabyte hole of zeros, or camera poses composed with a reversed rotation, and the summary read at the end still parses fine either way. Before generating all 103 files, decode at least one message per channel from the first one and eyeball a real value:
Only once that one episode looks right — ideally opened in a viewer, not just decoded in Python — is it worth calling build_episode() for the rest. Every trajectory name lives as a key in scene_data.json’s "trajectories" dict, so walking the five scene folders from Step 1 and that dict’s keys is enough to drive all 103 calls:
That’s the full pipeline: 5 scene folders in, 103 .mcap files out, one per drone flight, laid out as episodes/<scene>/<trajectory>.mcap for Step 3 to glob.
Step 3: Load the MCAP episodes into a FiftyOne dataset
Each .mcap file is one sample. FiftyOne reads the file extension and sets media_type to "multimodal" automatically:
Open a sample in the App, and you get a synchronized 3D view: the point cloud sitting still, the measured-pose camera frustum moving along its real flight path, and, on every frame COLMAP registered, a second frustum tracking the optimized pose next to it.
Because a trajectory is already a time series: one camera moving through one fixed map, with more than one candidate pose per moment. A flat image dataset can hold pose values as metadata, but it can’t play them back in sync with a shared 3D map the way an MCAP episode does in the FiftyOne/Foxglove viewer.
They keep their measured pose and lose nothing else. church/rocket_upward has zero registered frames dataset-wide, a documented issue in PIVOT’s own dataset card, not a bug in this pipeline. Its MCAP has a camera frustum moving throughout the entire flight, with no camera_colmap frustum at all.
Only if you’re building all 103 trajectories, not just the standard-FOV ones this post walks through. PIVOT’s wide-FOV calibration is a genuine k1-k4 fisheye fit, and it can turn non-monotonic (non-invertible) near the edge of the frame. Worth knowing before you hit an “invalid projection domain” error, but a distortion-model problem, not an MCAP-authoring one.
Roughly 28GB across 103 files for all five scenes, smaller than PIVOT’s own 56GB Hugging Face listing since the raw COLMAP intermediates never get downloaded in the first place.
