FiftyOne brings multimodal data curation and model evaluation to PyTorch workflows.
FiftyOne, Voxel51's open source multimodal data platform for physical AI, has been officially accepted into the
PyTorch Ecosystem.
The recognition reflects what many machine learning teams are already doing in practice: using PyTorch to build models and FiftyOne to visualize, curate, and evaluate the multimodal data those models depend on. As computer vision and multimodal AI systems become more capable, improving model quality increasingly depends on improving the data behind them.
What is the PyTorch Ecosystem?
The PyTorch Ecosystem is a curated collection of open source projects that extend and complement PyTorch. Projects are reviewed by the PyTorch Ecosystem Working Group for quality, ongoing maintenance, and value to the community before being accepted.
FiftyOne is now listed on the
PyTorch Landscape under Modeling → Multimodal, alongside other projects that help developers build production machine learning systems. FiftyOne brings data curation, visualization, and model evaluation into the PyTorch workflow, helping developers understand both their datasets and their models.
Why multimodal data matters for physical AI
Modern physical AI systems, from autonomous vehicles and robotics to smart infrastructure and industrial automation, learn from enormous collections of multimodal sensor data. Images, video, LiDAR, radar, and other sensor streams all contribute to how these systems perceive and interact with the physical world.
PyTorch provides the foundation for training many of today's multimodal machine learning models. FiftyOne helps developers understand the data that determines how well those models perform. With FiftyOne, teams can visualize
multimodal datasets,
curate training data,
annotate samples, and
evaluate model predictions to find the failure modes, annotation issues, and edge cases that aggregate metrics hide.
How FiftyOne works with PyTorch
FiftyOne and PyTorch have worked well together for years. Joining the PyTorch Ecosystem formalizes that relationship and makes it easier for developers to discover the integrations that already exist.
A typical workflow looks like this:
- Train or load a PyTorch model.
- Run inference on images, video, or grouped multimodal datasets in FiftyOne.
- Visualize predictions, embeddings, and model outputs.
- Identify failure modes, annotation issues, and rare edge cases.
- Curate or relabel data.
- Train and evaluate the next version of your model.
FiftyOne supports that workflow through several native integrations:
PyTorch Hub integration
Load pre-trained models directly from PyTorch Hub and run inference, generate embeddings, and evaluate predictions on your FiftyOne datasets. Learn more in the
PyTorch Hub integration docs.
For example, you can load a model from PyTorch Hub and run inference on a dataset in just a few lines:
The same pattern works for embeddings. Load a model like DINOv2 with an embeddings layer specified, then call compute_embeddings() to power similarity search and embeddings visualization in FiftyOne:
Native PyTorch data loading
Use FiftyOne datasets directly inside PyTorch training pipelines without maintaining separate dataset definitions. Any FiftyOne dataset or view can be converted into a torch.utils.data.Dataset with a single call to to_torch(). You define a GetItem that declares which fields your training loop needs and how to turn a sample into a model input:
Because the conversion works on any view, you can curate a training split in FiftyOne, filter out bad annotations, and feed exactly that slice of data to PyTorch, with no intermediate export step.
When data loading becomes the bottleneck, pass
vectorize=True to to_torch(). FiftyOne caches the required fields in memory upfront, so retrieving each sample during training is a simple lookup with no database reads in the hot path.Check out the
torch dataset recipes for complete training examples, including multiprocessing and distributed training.
Model evaluation
Evaluate object detection, classification, segmentation, and other visual AI tasks across images, video, and
grouped multimodal datasets, then interactively explore exactly where models succeed and where they fail.
Once your PyTorch model has made predictions, evaluating it takes one line, and every true positive, false positive, and false negative is recorded on the samples themselves:
Aggregate metrics tell you how your model is doing. The interactive part tells you why. Sorting, filtering, and viewing individual failures is where annotation mistakes and edge cases surface. For a complete walkthrough, see the
detection evaluation tutorial.
This extends naturally to physical AI. Here is an end-to-end sensor fusion scenario using a grouped dataset that pairs stereo camera images with LiDAR point clouds for each scene:
In the FiftyOne App, each group is displayed together: flip between camera views, inspect the 3D point cloud in the built-in 3D visualizer, and see exactly where your model succeeds and fails across modalities.
What's next
Joining the PyTorch Ecosystem is a milestone, but it's also the beginning of a closer relationship with one of the largest open source machine learning communities. We're excited to continue building alongside the PyTorch community, contributing to the open source machine learning ecosystem, and making it even easier for developers to move from training models to understanding and improving the multimodal data those models rely on.
Get started with FiftyOne and PyTorch
Thanks to the PyTorch Ecosystem Working Group for the warm welcome. We look forward to seeing what the PyTorch community builds with FiftyOne.
