FiftyOne Computer Vision Tips and Tricks – Nov 24, 2023

Welcome to our weekly FiftyOne tips and tricks blog where we recap interesting questions and answers that have recently popped up on Slack, GitHub, Stack Overflow, and Reddit.

As an open source community, the FiftyOne community is open to all. This means everyone is welcome to ask questions, and everyone is welcome to answer them. Continue reading to see the latest questions asked and answers provided!

Wait, what’s FiftyOne?

FiftyOne is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.

• If you like what you see on GitHub, give the project a star.
• Get started! We’ve made it easy to get up and running in a few minutes.
• Join the FiftyOne Slack community, we’re always happy to help.

Ok, let’s dive into this week’s tips and tricks!

Retrieving the filepaths of samples that are not part of a view

Community Slack member Mareeta asked:

Is there a way to get the filepaths of all the samples in a dataset that are not part of a view?

Yes! Tweak to taste the following:

filepaths = dataset.exclude(uniq_view).values("filepath")

Clustering images based on similarity

Community Slack member Mareeta asked:

Is there a function that allows for the clustering of images based on their similarity?

Yes! The FiftyOne Brain’s compute_visualization() method enables you to generate low-dimensional representations of the samples and/or individual objects in your datasets. There are four basic ways to generate embeddings using the embeddings and model parameters:

• Provide nothing, in which case a default general purpose model is used to embed your data
• Provide a Model instance or the name of any model from the Model Zoo that supports embeddings
• Provide your own precomputed embeddings in array form

Provide the name of a VectorField or ArrayField of your dataset in which precomputed embeddings are stored

Here’s an example that uses this FiftyOne Brain method with KMeans from sklearn to cluster your images based on similarity:

import fiftyone as fo
import fiftyone.zoo as foz

# Load mnist as an example dataset
dataset = foz.load_zoo_dataset("mnist", split="test", dataset_name="cluster_colors")

# Compute embeddings on the images (this took ~3 minutes on my laptop)
model = foz.load_zoo_model("mobilenet-v2-imagenet-torch")
embeddings = dataset.compute_embeddings(model)

# Compute a visualization of the embeddings
import fiftyone.brain as fob
results = fob.compute_visualization(dataset, embeddings=embeddings, brain_key="vis")

# Find clusters and store corresponding cluster labels
# Cluster on embeddings
from sklearn.cluster import KMeans
clusters = KMeans(n_clusters=10).fit_predict(embeddings)
dataset.set_values("embedding_cluster_label", clusters)

# Cluster on 2d points from after the dimensionality reduction in compute_visualization()
point_clusters = KMeans(n_clusters=10).fit_predict(results.points)
dataset.set_values("point_cluster_label", point_clusters)

# Visualize the cluster colors in the App
session = fo.launch_app(dataset)

# Use ground truth labels to find which cluster corresponds to which label
from fiftyone import ViewField as F
point_cluster_to_gt = {}
for lab in dataset.distinct("ground_truth.label"):
    d = dataset.match(F("ground_truth.label") == lab).count_values("point_cluster_label")
    pcl = max(d, key=d.get)
    gtl = int(lab[0])
    point_cluster_to_gt[pcl] = gtl
    
dataset.set_field("point_cluster_label", F().map_values(point_cluster_to_gt))
dataset.clone_sample_field("point_cluster_label", "remapped_cluster")
dataset.set_field("remapped_cluster", F("remapped_cluster").map_values(point_cluster_to_gt)).save()

# Evaluate the clustering performance vs ground truth
dataset.match(F("ground_truth.label").rsplit(" ")[0].to_int() != F("remapped_cluster")).set_field("ground_truth.eval", "fp").save()
dataset.add_dynamic_sample_fields()
# You can now filter your samples by the `fp` attribute of your `ground_truth` field in the App to see errors

Pick a few extra tips and tricks in this blog exploring embeddings in FiftyOne.

Filtering with dynamic attributes

Community Slack member Ran asked:

I have samples with “detections” (bounding boxes) in my dataset. These detections also have “attributes”. How can I filter by certain attribute value(s)?

Best bet is to use FiftyOne’s dynamic attributes. Any field(s) of your FiftyOne datasets that contain DynamicEmbeddedDocument values can have arbitrary custom attributes added to their instances. To see what attributes you already have that are available to declare, use:

print(dataset.get_dynamic_field_schema())

Then follow it up with add_sample_field()to declare any of them you wish:

dataset.add_sample_field("ground_truth.detections.iscrowd", fo.FloatField)

Computing pairwise IoUs between predicted and ground truth objects

Community Slack member Nahid asked:

In my image dataset, I have people and vehicle annotations in YOLOv5 format. Is there a way in FiftyOne to discover images where there is no overlap of these objects (and some distance between them)?

Yes! You could do this using FiftyOne’s compute_ious function which computes the pairwise IoUs between the predicted and ground truth objects. For example with two classes:

import fiftyone.utils.iou as foui

class_1 = dataset.filter_labels(field_name, F("label") == class1, only_matches=False).values(f"{field_name}.detections")
class_2 = dataset.filter_labels(field_name, F("label") == class2, only_matches=False).values(f"{field_name}.detections")
sample_ids = dataset.values("id")

non_overlapping_ids = []
for sample_id, class_1_dets, class_2_dets in zip(class_1, class_2, sample_ids):
    if class_1_dets and class_2_dets:
        if foui.compute_ious(class_1_dets, class_2_dets).any():
            # There exists at least one overlapping class 1 and class 2 detection in this sample
            continue
    non_overlapping_ids.append(sample_id)

non_overlapping_view = dataset.select(non_overlapping_ids)

Converting CVAT coordinates into FiftyOne compatible ones

Community Slack member Queenie asked:

I have an image with a width and height of 720 and 1280:

<image id="9" name="000009_horse_000143.png" width="720" height="1280"> 

The position of a bounding box in the image is set in CVAT to:

<box label="horse" source="file" occluded="0" xtl="12.00" ytl="521.00" xbr="538.00" ybr="915.00" z_order="0">

The corresponding coordinates in FiftyOne are:

[
   0.01666666753590107,
   0.40703123807907104,
   0.730555534362793,
   0.30781251192092896
]

What is the most efficient way to convert CVAT coordinates into FiftyOne compatible ones?

Because you are importing directly from CVAT into FiftyOne, the easiest way to accomplish this is to use FiftyOne’s native CVAT integration. FiftyOne provides an API to create tasks and jobs, upload data, define label schemas, and download annotations using CVAT, all programmatically in Python.

With the import_annotations() utility, you can import individual task(s) or an entire project into a FiftyOne dataset. The conversion of coordinates between CVAT and FiftyOne will be taken care of as part of the import. You can find a code example here.

Join the FiftyOne Community!

Join the thousands of engineers and data scientists already using FiftyOne to solve some of the most challenging problems in computer vision today!

• 2,000+ FiftyOne Slack members
• 4,000+ stars on GitHub
• 5,000+ Meetup members
• Used by 370+ repositories
• 60+ contributors