BabyROS and FiftyOne: See What Your Robot Gets Wrong

Robots generate an absurd amount of data — camera frames, detections, point clouds, all day, every day. Most of it disappears into a log file nobody opens until something breaks. That's the gap this post is about closing.

What Is BabyROS? A lightweight ROS alternative built on Zenoh

BabyROS from the folks over at Telekinesis is ROS without the ceremony. It's a lightweight pub/sub layer built on the Zenoh protocol: one process shouts a message onto a topic, any other process listening picks it up. No massive install, no rigid message schema baked into the framework, no waiting on a build system. If you've ever thought "I just need two processes to talk to each other, why is this so hard" — that's the itch BabyROS scratches.

What it deliberately doesn't do is tell you what your messages mean. That's a feature, not a gap — but it means you need to bring your own contract.

Key Takeaways

BabyROS moves the data and FiftyOne makes it mean something. BabyROS is a lightweight pub/sub transport, and FiftyOne turns the frames, detections, and point clouds it carries into a dataset you can query, evaluate, and fix.
A fixed schema up front does the heavy lifting. Define a contract for your messages (Telekinesis Data Engine types work well) and the transport never has to know what it carries, while FiftyOne on the other end knows exactly what to do with it.
Curation happens live, not in a postmortem. Samples land in a FiftyOne dataset as the robot runs, so you catch a systematic mistake in the moment instead of three weeks later.
One pattern covers 2D and 3D, batch and live. Boxes on pixels or cuboids on point clouds, a one-time load or a stream that keeps growing, the same transport and contract carry all of it.
The whole thing runs in one notebook. Publish real perception data over BabyROS, inject realistic detector mistakes, run mAP evaluation, go 3D, and stream live, all in a single walkthrough you can run today.

FiftyOne: the difference between a metric and an explanation

FiftyOne is an open source library that makes a pile of images, detections, and embeddings into something a human can actually reason about. Load a dataset, and suddenly you can filter by confidence, tag false positives, run mAP evaluation, visualize embeddings, and — if you're feeling ambitious — orbit a 3D point cloud in your browser. It's the difference between "the model is 89% accurate" and "here are the 11 specific frames it got wrong, and here's why."

How BabyROS and FiftyOne work together for live curation

BabyROS moves bytes. FiftyOne makes sense of them. Alone, each is useful. Together, something better happens:

Messages become legible. A dict arriving in a subscriber callback is mute. The same dict rendered as an image with boxes drawn on it is a conversation.

The contract does the heavy lifting. Give your messages a fixed schema up front (Telekinesis's Data Engine types work well here) and the transport layer never has to know what it's carrying — FiftyOne on the other end just knows what to do with it.

Curation stops being a batch job. Instead of exporting logs at the end of a shift, samples land in a FiftyOne dataset as they happen. You can catch a systematic mistake while the robot is still running, not three weeks later during a postmortem.

One pipeline, 2D or 3D, batch or live. Boxes on pixels, cuboids on point clouds, a one-time load or a dataset that just keeps growing — same transport, same contract, same curation habits either way.

The team gets a shared view. With FiftyOne Enterprise it’s a URL. Anyone can open it and see the same dataset, instead of squinting at whoever's terminal happens to be running the subscriber.

Build a BabyROS + FiftyOne perception pipeline (notebook walkthrough)

We put together a notebook to simulate all these pieces working together that goes further than you'd expect from a "hello world" example:

Publishes real perception data over BabyROS, using Telekinesis's Data Engine types (ObjectDetectionAnnotation, Category) as the message schema — either a real, photorealistic Kaggle bin-picking dataset, or a zero-setup synthetic fallback if you just want to see it run.

Ships two detector modes: one that perturbs ground truth for clean, tunable demo numbers, and one that's a genuinely real detector — Hough-circle detection running on actual pixels, no ground-truth peeking.

Injects the mistakes that make curation worth doing: missed detections, duplicate boxes, hallucinated boxes — so there's something real for FiftyOne to find.

Runs full evaluation, mAP and per-class precision/recall included, straight out of evaluate_detections — because the schema was right from the start.

Goes 3D: the same transport-and-contract pattern, but with Boxes3D/Points3D and point clouds instead of pixels.

Goes live: a background thread feeds the FiftyOne dataset one sample at a time, indefinitely, until you tell it to stop — the closest thing to watching a robot think in real time.

Saves views, so the App remembers "show me the false positives" instead of you re-typing the filter every time.

Next steps: docs, datasets, and how to build it yourself

Read the BabyROS docs: docs.telekinesis.ai/babyros
Browse the BabyROS source: github.com/telekinesis-ai/babyros
Read the Data Engine docs (the schema layer that makes messages self-describing): docs.telekinesis.ai/data-engine
Explore FiftyOne: docs.voxel51.com
Grab real bin-picking datasets to try this on: kaggle.com/telekinesisai — synthetic, COCO-annotated, free
Build the pipeline yourself: publish over a BabyROS topic, shape your messages with a fixed schema, subscribe straight into a fo.Dataset. That's the whole pattern.

FAQ

What is BabyROS?

BabyROS is a lightweight publish/subscribe messaging layer for robotics, built on the Zenoh protocol. One process publishes a message to a topic and any process listening picks it up, with no heavy install, rigid schema, or build system required. Telekinesis maintains it as a stripped-down alternative to ROS for teams that just need two processes to talk.

Is BabyROS a replacement for ROS?

BabyROS covers the core pub/sub pattern most teams reach ROS for, without the ceremony of a full ROS install. It deliberately leaves out message semantics, so you bring your own schema to define what messages mean. For lightweight perception pipelines and prototyping it stands on its own, while large robotics stacks may still want full ROS.

What does FiftyOne do with robot perception data?

FiftyOne is a data platform for visual and multimodal AI that turns images, detections, embeddings, and point clouds into a dataset you can explore and evaluate. You can filter by confidence, tag false positives, run mAP with per-class precision and recall, visualize embeddings, and orbit 3D point clouds in the browser. It takes you from a single accuracy number to the exact frames a model got wrong.

How does the BabyROS and FiftyOne integration work?

BabyROS handles transport and FiftyOne handles curation. You publish perception data over a BabyROS topic using a fixed schema, then a subscriber loads each message straight into a FiftyOne dataset. Because the schema is defined up front, FiftyOne knows how to render boxes, cuboids, and point clouds without the transport layer understanding the payload. The result is a dataset that fills up as the robot runs.

Can this handle 3D point clouds?

Yes. The same transport and contract pattern that carries 2D boxes on pixels also carries Boxes3D and Points3D with point clouds. You publish 3D perception data over BabyROS, load it into FiftyOne, and orbit the scene in the browser. Curation, evaluation, and saved views work the same way in 3D as they do in 2D.

Do I need real robot data to try it?

No. The demo notebook ships a zero-setup synthetic fallback, so you can watch the full pipeline run without any data of your own. When you are ready for realistic data, it also works with photorealistic, COCO-annotated bin-picking datasets from Telekinesis on Kaggle.

Robots Ship Silence About Mistakes. BabyROS and FiftyOne Fix That.

Talk to an AI expert