TII-RATM Drone Racing in FiftyOne: When Onboard Odometry Never Recovers

Key takeaways

• In four of six TII-RATM drone racing flights, visual-inertial odometry (VIO) records its single worst disagreement with motion capture ground truth on the very last pose of the flight, with no recovery afterward.
• Dividing the tracking error at the final pose by the maximum error anywhere in the run, "final over max," separates a VIO system that recovers from drift from one that ends on its worst moment. RMSE alone cannot tell them apart.
• The worst flight, flight-07p-lemniscate, reaches 1.66 meters RMSE, but the flight that actually recovers, flight-03p-ellipse, has a higher RMSE than one that never recovers.
• Across all six flights, the onboard fisheye feature tracker produces usable matches on only 53.9% to 57.9% of camera frames, a structural cost of flying a wide fisheye lens fast indoors that tracking error alone never surfaces.
• FiftyOne 1.22.0 ships tii-ratm-drone-racing in the Dataset Zoo as native MCAP episodes, so both trajectories render in the multimodal viewer with no conversion step and no model training.

FiftyOne 1.22.0 added a real onboard-vs-ground-truth racing benchmark

Two numbers per timestamp: visual-inertial odometry versus motion capture ground truth

Six flights, three tiles each, no model training required

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("tii-ratm-drone-racing")
worst_first = dataset.sort_by("tracking_error_rmse_m", reverse=True)

session = fo.launch_app(worst_first)

The worst flight, in FiftyOne's own 3D viewer, both trajectories at once

What we measured: does the drift ever recover?

A second, independent failure: the fisheye tracker drops nearly half of every frame

Building your own

• Compute final-over-max, not just RMSE. A single summary error number hides whether a system recovers from drift or ends on its worst moment. Dividing the error at the last pose by the largest error anywhere in the run turns that into one comparable number per run.
• Sort before you look. dataset.sort_by("tracking_error_rmse_m", reverse=True) put the flight worth opening first, out of six, without watching any video.
• Check coverage fields, not just accuracy fields. num_feature_track_frames next to num_camera_frames caught a real sensor limitation that tracking error alone never would have shown, since the system compensated for it well enough to still finish the lap.
• Let the viewer show what the query found. Once the query has picked the flight and the moment, FiftyOne's multimodal viewer draws both trajectories in the same 3D space for free, no custom plotting code to write.

Where does vision-only localization matter outside drone racing?

• Indoor and warehouse robots with no GPS. A forklift-replacement AMR or a warehouse inspection drone loses GPS the moment it goes indoors, and adding a motion capture rig or a fleet of UWB anchors to every building it visits is not realistic. Vision-inertial odometry is often the only localization it gets, and knowing whether that estimate corrects itself after a rack aisle or a low-light dock, versus committing to its worst error right at the docking maneuver, is the difference between a near miss and a collision report.
• Autonomous drone racing and inspection flights near obstacles. This dataset is itself drawn from that world: a racing quadrotor flying gates at speed on a single fisheye camera. The same sensor and the same odometry stack show up in infrastructure inspection drones flying close to towers, turbines, or bridge undersides, where the final approach to the structure is exactly when a lingering drift is most expensive.
• Any vision-only SLAM stack evaluated only on RMSE. RMSE and median error are the numbers most benchmark leaderboards report, and both of those flights that never recover would look unremarkable on that alone (flight-02p-ellipse has a lower RMSE, 0.63 m, than the flight that actually corrects itself). A team picking a SLAM stack off a leaderboard number alone would miss exactly the failure mode this check is built to catch.

Try it