Welcome to our weekly FiftyOne tips and tricks blog where we give practical pointers for using FiftyOne on topics inspired by discussions in the open source community. This week we’ll cover some tips and tricks that will help you accelerate your computer vision workflows using FiftyOne.
Wait, What’s FiftyOne?
FiftyOne is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.
[@portabletext/react] Unknown block type "externalImage", specify a component for it in the `components.types` prop
Ok, let’s dive into this week’s tips and tricks!
Load data into FiftyOne faster
One of the great features of PyTorch is the
DataLoader
class, which makes it easy to
efficiently load and process data. This becomes especially useful when dealing with large datasets, where it can be inefficient or impossible to load the entire dataset into memory at once.
When you’re working with PyTorch data in FiftyOne, you can leverage the speed and simplicity of your PyTorch DataLoader
to import your data into and analyze that data in FiftyOne.
First, we create the TorchVision dataset and instantiate a DataLoader
for the dataset.
Then we create the FiftyOne Dataset
for this data and specify the directory in which we will store the images.
Finally, we create a sample parser and use the PyTorch DataLoader
with FiftyOne’s ingest_labeled_images()
method to fill our FiftyOne Dataset
with the PyTorch data.
Once you have PyTorch data loaded into FiftyOne, it is also easy to train
Lightning Flash tasks on your FiftyOne data and add model predictions to FiftyOne!
Peruse through samples faster
If you’re working with high-resolution media files like satellite imagery or photo-realistic AI-generated data, you might notice some slight delays in rendering when you scroll through samples in the sample grid in the FiftyOne App. This is because, by default, the app is rendering the high-resolution media files in real time.
If you’re experiencing this buffering, you might get a speed boost by configuring the app to use thumbnail images! By modifying the App Config, you can enable
multiple media fields and choose which media field is displayed in the sample grid. One common workflow is creating lower resolution, downscaled images for each sample, and configuring the app to display these in the sample grid.
This way you retain the depth of information in your high-resolution media files for downstream workflows without sacrificing speed while investigating your data. When you click on a thumbnail in the sample grid, the media file in the resulting full-screen modal will be the high-resolution version!
Here’s an example of how you might accomplish this effect:
First, we create thumbnail images and store their paths in a thumbnail_path
field:
Then we modify the
dataset’s App config to expose these thumbnails in the sample grid and create a session:
Filter samples faster
Another way to accelerate visually inspection of your data in the FiftyOne App is by setting the sidebar mode to
”fast”
. When filtering and matching via the sidebar in the App, the
sidebar_mode
property in the
App Config allows you to specify whether these operations should be applied to all samples in the view and all of the relevant fields, or just the samples visible in the sample grid and fields that are expanded in the filter tray. This can be a dramatic time-saver for datasets with many samples, many fields, or both.
For datasets with more than 10,000 samples, the FiftyOne App’s default behavior is ”fast”
, but if you want to make this the default for all of your datasets, you can set this in the App Config with:
Alternatively, if you want to set this only for a specific dataset, you can modify the dataset’s app config:
Start with most unique samples
In many machine learning workflows, some samples matter more than others. Whether you are deciding what subsets of your dataset to send out for annotation, or identifying edge cases and failure modes of your models, it helps to focus your attention.
One trick you can use to explore a broad range of samples rather than scrolling through a bunch of similar examples is to look at the most “unique” samples first.
With the FiftyOne Brain, you can use embeddings to compute a score for each sample in your dataset that tells you how unique that sample is, relative to all of the other samples.
You can then sort your samples by uniqueness, passing in reverse=True
so that the most unique samples (highest values in uniqueness
field) appear first:
Then you can save time by looking at the most unique samples in your dataset first!
Pre-annotate with embeddings
Embeddings can also save you precious time by helping you pre-annotate your computer vision data. Labeling and annotation of ground truth data often represents one of the most tedious and time-intensive tasks in computer vision workflows. In addition to hours or days spent, annotation can also be quite expensive.
Fortunately, for some datasets it is possible to use the structure uncovered by embeddings to do a coarse first pass through the data.
With the
MNIST dataset, for instance, computing embeddings and then using dimensionality reduction techniques like
tSNE, you can uncover multiple mostly-distinct clusters of samples. Using the lasso to interact with this visualization and view each cluster in the FiftyOne App, it quickly becomes clear that these clusters roughly correspond to different numerals. The MNIST dataset of course already has ground truth labels, but if it didn’t, you could leverage these clusters to tag samples with pre-annotations.
Join the FiftyOne community!
Join the thousands of engineers and data scientists already using FiftyOne to solve some of the most challenging problems in computer vision today!