Eight Sensors, One Dataset, Zero Blind Spots: Exploring OctoSense with FiftyOne
Jul 10, 2026
• 6 min read
In this article
In this article
Talk to an AI expert
Robotics data is famously hard to look at. It arrives as rosbags full of LiDAR sweeps, event streams, IMU traces, and video — time-synced, multi-gigabyte, and allergic to a simple image viewer. So most teams fly blind, training on data they've never actually inspected. Let's fix that.
Meet OctoSense: A Robot's-Eye View of the World
OctoSense is a beast of a multimodal dataset — hundreds of time-synchronized driving sequences captured across cars, quadrupeds, and marine platforms. Every sequence bundles stereo RGB, stereo event cameras, infrared, a 64-beam LiDAR, IMU, GPS, and CAN-bus data, all locked to a single PPS-synchronized clock. On top of the raw sensors it ships ground truth for depth, semantic segmentation, optical flow, odometry, and scene captions.
In short: it's one of the richest open sensor suites you can get your hands on — and a perfect stress test for any data tool that claims to handle "multimodal."
| Layer | Why it matters in FiftyOne |
|---|---|
| Stereo RGB frame | Base frame that every label aligns to |
| LiDAR depth (GT) | Shown as a heatmap to hunt holes and blooming artifacts |
| Semantic segmentation (GT) | Side-by-side label QA |
| Optical flow (GT) | Reads forward motion straight off the pixels |
| Scene captions | Powers caption-embedding search |
| Per-frame metadata | Filter and slice frames by driving conditions |
Key Takeaways
- OctoSense is a multimodal robotics dataset of hundreds of time-synchronized driving sequences across cars, quadrupeds, and marine platforms, bundling stereo RGB, stereo event cameras, infrared, 64-beam LiDAR, IMU, GPS, and CAN-bus data on a single PPS-synchronized clock.
- It ships ground truth for depth, semantic segmentation, optical flow, odometry, and scene captions.
- FiftyOne turns the raw sensor archive into a single browsable, filterable, and searchable dataset, so you can QA labels, cluster scenes, and curate slices across every sequence at once.
- Because OctoSense renders all ground truth into the rectified left-camera frame, one FiftyOne sample holds a frame with every label pixel-perfectly aligned to it.
- The demo builds two search indexes: a CLIP index over pixels for natural-language visual search, plus OctoSense's precomputed caption embeddings for description-based search.
- Stacking depth, segmentation, and flow on one frame surfaces label errors, and toggling ego-motion optical flow alone makes the car's forward motion visible as a radial magnitude gradient.
Meet FiftyOne: X-Ray Vision for Your Dataset
FiftyOne is the open-source toolkit for curating and understanding visual datasets. It gives you a fast, filterable App to browse samples, overlay labels, cluster embeddings, search by natural language, and slice your data by any field you can dream up.
It's the difference between having a dataset and actually knowing it.
Why OctoSense + FiftyOne = ❤️
OctoSense already ships per-sequence tooling — a Rerun timeline viewer, and a caption-search CLI. What it doesn't have is a way to explore the whole collection at once: to filter every frame by driving conditions, QA the labels side by side, cluster the scenes, and search across sequences — all cross-linked in one view.
That's exactly the hole FiftyOne fills. Point it at OctoSense and a raw sensor archive becomes a browsable, queryable, curatable dataset. Rich data meets the cockpit built to fly it.
Demo: From Rosbags to a Browsable Dataset
Our demo notebook turns OctoSense sequences into a fully loaded FiftyOne dataset. The clever bit: because OctoSense renders all its ground truth into the rectified left-camera frame — and hands you the exact video-frame index for every LiDAR scan — one FiftyOne sample can hold a frame with every label pixel-perfectly aligned to it. So each sample carries:
- Semantic segmentation masks (Cityscapes-19) for label QA
- LiDAR depth as a heatmap — hunt for holes and blooming artifacts
- Ego-motion optical flow, derived at ingest from depth + poses
- Scene captions from a vision-language model
- Per-frame + per-sequence metadata — speed, turn rate, day/night, sensor dropout, GPS
Show Me, Don't Tell Me: Search, Cluster, and Curate OctoSense
Two brains, two kinds of search. We build a CLIP index over the pixels for natural-language visual search ("pedestrian crossing," "car turning left") and load OctoSense's own precomputed caption embeddings as a second index — so you can also find frames whose scene description matches. Same dataset, two complementary ways to ask "show me more like this."
Lasso the latent space. An embedding plot clusters the whole dataset in 2D. Draw a loop around a cluster and watch those frames light up in the grid — day/night and highway/residential tend to separate on their own.
Curate like a pro. The notebook ships with named views ready in the App's dropdown — "Flow demo: fast-moving," "QA: seg + moving," "Prune: near-stationary," "Sharp turns" — so you switch between curation slices with one click. Build a hard-case review set of fast, sharply-turning, seg-labeled frames, or isolate near-stationary stop-and-go frames as prune candidates. Dataset curation as a point-and-click activity, not a scripting chore.
QA labels that lie. Stack depth, segmentation, and flow on a single frame and the artifacts jump out — mask bleed at object edges, depth dropouts around signs, and flow that disagrees with a moving vehicle (because the depth GT masked it as dynamic). One label reveals how another was built.
Reading Motion Straight Off the Pixels
Toggle on ego-motion optical flow alone and something beautiful happens. The flow field renders as a magnitude colormap — and it maps perfectly onto the physics of driving forward. The fast-moving road surface right in front of the car lights up bright yellow-green: those close points stream past fast. The vanishing point down the street stays deep blue: far-away points barely move. Between them, a smooth radial gradient spreading out from the focus of expansion.
That pattern isn't decoration — it's the car's forward motion, made visible. You can read the ego-motion off the colors. And when a genuinely moving object enters the scene, it breaks the gradient — which is exactly how you'd spot it in a QA pass. It's the kind of figure that makes a dataset click for someone who's never touched a rosbag.
Where to Go Next
- Grab the dataset: OctoSense on Hugging Face
- Get the code: OctoSense on GitHub
- Learn FiftyOne: Docs · FiftyOne Brain · GitHub
- Demo Notebook: GitHub
Scale it up. Add more sequences (mix in night drives — the notebook auto-skips segmentation where it's unavailable), tighten the sampling stride for denser coverage, and swap the similarity backend for a real vector database when your collection grows into the thousands. Then bring in the modalities we left on the bench — the event cameras and full LiDAR point clouds — and see how much more of the robot's world you can put under glass.
FAQ
What is OctoSense?
OctoSense is a multimodal robotics dataset made up of hundreds of time-synchronized driving sequences captured across cars, quadrupeds, and marine platforms. Each sequence bundles stereo RGB, stereo event cameras, infrared, a 64-beam LiDAR, IMU, GPS, and CAN-bus data locked to a single PPS-synchronized clock, plus ground truth for depth, semantic segmentation, optical flow, odometry, and scene captions.
What does FiftyOne add to OctoSense?
OctoSense already ships per-sequence tooling like a Rerun timeline viewer and a caption-search CLI. FiftyOne adds a way to explore the whole collection at once: filter every frame by driving conditions, QA labels side by side, cluster scenes, and search across sequences, all cross-linked in one view.
How does one FiftyOne sample hold so many labels?
OctoSense renders all of its ground truth into the rectified left-camera frame and provides the exact video-frame index for every LiDAR scan. That lets a single FiftyOne sample carry segmentation masks, LiDAR depth, optical flow, scene captions, and per-frame metadata all pixel-perfectly aligned to one frame.
How does the demo support natural-language search?
The notebook builds two indexes. A CLIP index over the pixels powers natural-language visual search like "pedestrian crossing" or "car turning left," and OctoSense's precomputed caption embeddings act as a second index so you can find frames whose scene description matches your query.
How can you tell good labels from bad ones?
Stacking depth, segmentation, and optical flow on a single frame makes artifacts jump out, such as mask bleed at object edges, depth dropouts around signs, and flow that disagrees with a moving vehicle. One label reveals how another was built.
Loading related posts...
© 2026 Voxel51 All Rights Reserved
