FiftyOne Joins the PyTorch Ecosystem

FiftyOne brings multimodal data curation and model evaluation to PyTorch workflows.

FiftyOne, Voxel51's open source multimodal data platform for physical AI, has been officially accepted into the PyTorch Ecosystem.

The recognition reflects what many machine learning teams are already doing in practice: using PyTorch to build models and FiftyOne to visualize, curate, and evaluate the multimodal data those models depend on. As computer vision and multimodal AI systems become more capable, improving model quality increasingly depends on improving the data behind them.

What is the PyTorch Ecosystem?

The PyTorch Ecosystem is a curated collection of open source projects that extend and complement PyTorch. Projects are reviewed by the PyTorch Ecosystem Working Group for quality, ongoing maintenance, and value to the community before being accepted.

FiftyOne is now listed on the PyTorch Landscape under Modeling → Multimodal, alongside other projects that help developers build production machine learning systems. FiftyOne brings data curation, visualization, and model evaluation into the PyTorch workflow, helping developers understand both their datasets and their models.

Why multimodal data matters for physical AI

Modern physical AI systems, from autonomous vehicles and robotics to smart infrastructure and industrial automation, learn from enormous collections of multimodal sensor data. Images, video, LiDAR, radar, and other sensor streams all contribute to how these systems perceive and interact with the physical world.

PyTorch provides the foundation for training many of today's multimodal machine learning models. FiftyOne helps developers understand the data that determines how well those models perform. With FiftyOne, teams can visualize multimodal datasets, curate training data, annotate samples, and evaluate model predictions to find the failure modes, annotation issues, and edge cases that aggregate metrics hide.

How FiftyOne works with PyTorch

FiftyOne and PyTorch have worked well together for years. Joining the PyTorch Ecosystem formalizes that relationship and makes it easier for developers to discover the integrations that already exist.

A typical workflow looks like this:

Train or load a PyTorch model.
Run inference on images, video, or grouped multimodal datasets in FiftyOne.
Visualize predictions, embeddings, and model outputs.
Identify failure modes, annotation issues, and rare edge cases.
Curate or relabel data.
Train and evaluate the next version of your model.

FiftyOne supports that workflow through several native integrations:

PyTorch Hub integration

Load pre-trained models directly from PyTorch Hub and run inference, generate embeddings, and evaluate predictions on your FiftyOne datasets. Learn more in the PyTorch Hub integration docs.

For example, you can load a model from PyTorch Hub and run inference on a dataset in just a few lines:

  import fiftyone.zoo as foz
  import fiftyone.utils.torch as fout
  dataset = foz.load_zoo_dataset("quickstart")
  # Load any model from PyTorch Hub
  model = fout.load_torch_hub_image_model(
      "pytorch/vision",
      "resnet18",
      hub_kwargs=dict(weights="ResNet18_Weights.DEFAULT"),
  )
  dataset.apply_model(model, label_field="resnet18")

The same pattern works for embeddings. Load a model like DINOv2 with an embeddings layer specified, then call compute_embeddings() to power similarity search and embeddings visualization in FiftyOne:

model = fout.load_torch_hub_image_model(
      "facebookresearch/dinov2",
      "dinov2_vits14",
      image_patch_size=14,
      embeddings_layer="head",
  )
  embeddings = dataset.compute_embeddings(model)

Native PyTorch data loading

Use FiftyOne datasets directly inside PyTorch training pipelines without maintaining separate dataset definitions. Any FiftyOne dataset or view can be converted into a torch.utils.data.Dataset with a single call to to_torch(). You define a GetItem that declares which fields your training loop needs and how to turn a sample into a model input:

import torch
  import fiftyone.utils.torch as fout
  from torchvision import transforms
  from PIL import Image
  class ClassificationGetItem(fout.GetItem):
      transform = transforms.Compose(
          [transforms.Resize((224, 224)), transforms.ToTensor()]
      )
      @property
      def required_keys(self):
          return ["filepath", "ground_truth"]
      def __call__(self, d):
          image = self.transform(Image.open(d["filepath"]).convert("RGB"))
          return image, d["ground_truth"].label
  torch_dataset = dataset.to_torch(ClassificationGetItem())
  data_loader = torch.utils.data.DataLoader(
      torch_dataset,
      batch_size=16,
      shuffle=True,
      num_workers=4,
      worker_init_fn=fout.FiftyOneTorchDataset.worker_init,
  )

Because the conversion works on any view, you can curate a training split in FiftyOne, filter out bad annotations, and feed exactly that slice of data to PyTorch, with no intermediate export step.

When data loading becomes the bottleneck, pass vectorize=True to to_torch(). FiftyOne caches the required fields in memory upfront, so retrieving each sample during training is a simple lookup with no database reads in the hot path.Check out the torch dataset recipes for complete training examples, including multiprocessing and distributed training.

Model evaluation

Evaluate object detection, classification, segmentation, and other visual AI tasks across images, video, and grouped multimodal datasets, then interactively explore exactly where models succeed and where they fail.

Once your PyTorch model has made predictions, evaluating it takes one line, and every true positive, false positive, and false negative is recorded on the samples themselves:

results = dataset.evaluate_detections(
      "predictions",
      gt_field="ground_truth",
      eval_key="eval",
      compute_mAP=True,
  )
  print(results.mAP())
  results.print_report()
  # Explore the samples with the most false positives in the App
  import fiftyone as fo
  session = fo.launch_app(dataset.sort_by("eval_fp", reverse=True))

Aggregate metrics tell you how your model is doing. The interactive part tells you why. Sorting, filtering, and viewing individual failures is where annotation mistakes and edge cases surface. For a complete walkthrough, see the detection evaluation tutorial.

This extends naturally to physical AI. Here is an end-to-end sensor fusion scenario using a grouped dataset that pairs stereo camera images with LiDAR point clouds for each scene:

 import fiftyone as fo
  import fiftyone.zoo as foz
  # Camera images + LiDAR point clouds, grouped by scene
  dataset = foz.load_zoo_dataset("quickstart-groups")
  # Run a PyTorch detection model on the left camera images
  model = foz.load_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")
  left_images = dataset.select_group_slices("left")
  left_images.apply_model(model, label_field="predictions")
  # COCO and KITTI use different label names; normalize before evaluating
  left_images = left_images.map_labels(
      "predictions", {"car": "Car", "person": "Pedestrian", "truck": "Truck"}
  )
  results = left_images.evaluate_detections(
      "predictions",
      gt_field="ground_truth",
      eval_key="eval",
  )
  # Browse predictions alongside the corresponding point clouds
  session = fo.launch_app(dataset)

In the FiftyOne App, each group is displayed together: flip between camera views, inspect the 3D point cloud in the built-in 3D visualizer, and see exactly where your model succeeds and fails across modalities.

What's next

Joining the PyTorch Ecosystem is a milestone, but it's also the beginning of a closer relationship with one of the largest open source machine learning communities. We're excited to continue building alongside the PyTorch community, contributing to the open source machine learning ecosystem, and making it even easier for developers to move from training models to understanding and improving the multimodal data those models rely on.

Get started with FiftyOne and PyTorch

Explore the PyTorch Hub integration docs
Star and contribute to FiftyOne on GitHub
Learn more about FiftyOne’s support for multimodal data in physical AI
Join the FiftyOne Discord community

Thanks to the PyTorch Ecosystem Working Group for the warm welcome. We look forward to seeing what the PyTorch community builds with FiftyOne.

Jesse Mostipak is the SEO and Content Manager at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn