Welcome to our weekly FiftyOne tips and tricks blog where we recap interesting questions and answers that have recently popped up on Slack, GitHub, Stack Overflow, and Reddit.
Wait, what’s FiftyOne?
FiftyOne is an open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.
- If you like what you see on GitHub, give the project a star.
- Get started! We’ve made it easy to get up and running in a few minutes.
- Join the FiftyOne Slack community, we’re always happy to help.
Ok, let’s dive into this week’s tips and tricks!
Evaluating performance by label
Community Slack member Mike Kraus asked,
“I have run a number of evaluations on the same dataset, and I want to compute evaluation metrics for each label separately. What is the best way to do this?
If you’re interested in composite metrics like the precision, recall, or f1-score, then look no further than FiftyOne’s evaluation API. The evaluate_detections() and evaluate_classifications() methods compute separate values for these metrics for each class, which can be viewed by calling print_report().
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
result = dataset.evaluate_detections(
"predictions", gt_field="ground_truth", eval_key="eval"
)
result.print_report()For more granular access to the detection counts, including false positives “fp” and false negatives “fn”, the evaluation API can be combined with FiftyOne’s filter() method to compute results by label, tag, or more flexible conditions.
To count the false positives and false negatives for each class, we can pick up where we left off above:
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
result = dataset.evaluate_detections(
"predictions", gt_field="ground_truth", eval_key="eval"
)
counting_results = {}
for detection_result in ["tp", "fp", "fn"]:
view = dataset.filter_labels("predictions", F("eval") == type)
counting_results[type] = view.count_values("predictions.detections.label")
print(counting_results)Learn more about FiftyOne’s Evaluation API in the FiftyOne Docs.
Exporting a dataset with splits
Community Slack member Joy Timmermans asked,
“Is it possible to export a dataset with splits in it?”
There are multiple ways to do this in FiftyOne. The first approach involves creating a separate view for each split you are interested in, and exporting each one of them. For instance, the following will export only the train and validation splits for a dataset:
export_dir_train = "/path/for/export/train"
export_dir_val = "/path/for/export/val"
# The name of the sample field containing the label that you wish to export
# Used when exporting labeled datasets (e.g., classification or detection)
label_field = "ground_truth" # for example
# The type of dataset to export
# Any subclass of `fiftyone.types.Dataset` is supported
dataset_type = fo.types.COCODetectionDataset
train_view = dataset.match_tags("train")
train_view.export(
export_dir=export_dir_train,
dataset_type=dataset_type,
label_field=label_field,
)
test_view = dataset.match_tags("val")
test_view.export(
export_dir=export_dir_val,
dataset_type=dataset_type,
label_field=label_field,
)Alternatively, you can leverage the “split” keyword argument as input to export(), which will take care of the splitting for you.
import fiftyone as fo
import fiftyone.zoo as foz
export_dir = "/path/for/export"
label_field = "ground_truth" # for example
# The splits to export
splits = ["train", "val"]
# All splits must use the same classes list
classes = ["list", "of", "classes"]
# The dataset or view to export
# We assume the dataset uses sample tags to encode the splits to export
dataset_or_view = foz.load_zoo_model("quickstart") # portion of COCO dataset
dataset_type = fo.types.COCODetectionDataset
# Export the splits
for split in splits:
split_view = dataset_or_view.match_tags(split)
split_view.export(
export_dir=export_dir,
dataset_type=fo.types.YOLOv5Dataset,
label_field=label_field,
split=split,
classes=classes,
)Learn more about exporting datasets in FiftyOne in the FiftyOne Docs.
Getting and setting values in large datasets
Community Slack member Geoffrey Keating asked,
“Is there a point at which calling the values() method on a very large dataset or field will be impractical?”
This question cuts to the core of what makes FiftyOne great for dealing with massive computer vision datasets. Whereas FiftyOne does provide a values() method for getting the values in a field, and the related set_values() method for setting a field’s values, these methods are typically only practical for small to medium sized datasets. This is because these two methods loads the entire dataset from memory at once, and the size of the dataset can exceed the amount of available RAM.
An alternative to use the select_values() and set_field() methods, which load in only constant memory. Of course, these methods do sacrifice speed in the name of conserving memory.
To efficiently do more general operations on a field of a large dataset, select_fields() and set_field() can be combined with iter_samples() to iterate through all samples. For instance, if we wanted to keep a running tally of all detections across all samples, and add this counter value in a field “detection_number” on each detection, we could do so with:
counter = 0
dataset.add_sample_field(
"ground_truth.detections.detection_number", fo.IntField
)
for sample in dataset.select_fields("ground_truth.detections").iter_samples(
autosave=True, progress=True
):
for detection in sample.ground_truth.detections:
detection.set_field("detection_number", counter)
counter += 1Learn more about set_field() and Fields in the FiftyOne Docs.
Storing group level metadata
Community Slack member Andreas Fehlner asked,
“I really like the new group feature. At the moment there is metadata common to all of the media in a group. Is it possible to store that metadata for all of the files in one group?”
One way to do this is to select one group slice and iterate through all samples in that slice. For each sample, you can then get all of the other samples in the same group using the get_group() method, and then add the metadata to all samples in the group.
left_samples = dataset.select_group_slices("left")
slices = dataset.group_slices
for ls in left_samples.iter_samples(autosave=True, progress=True):
group_metadata = ls.filepath ### Will be same for entire group
group_id = ls.group.id
group = dataset.get_group(group_id)
for s in slices:
dataset.group_slice = s
sample = dataset[group[s].id] ### Select from dataset by id
sample["group_metadata_field"] = group_metadataAlternatively, if you have multiple related media files that have the same aspect ratio, you can use multiple media fields on a single sample. For instance, if we want to generate low resolution thumbnails for each image in our dataset, we can tie these thumbnails to the original images in a single sample object.
import fiftyone as fo
import fiftyone.utils.image as foui
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
# Generate some thumbnail images
foui.transform_images(
dataset,
size=(-1, 32),
output_field="thumbnail_path",
output_dir="/tmp/thumbnails",
)
# Modify the dataset's App config
dataset.app_config.media_fields = ["filepath", "thumbnail_path"]
dataset.app_config.grid_media_field = "thumbnail_path"
dataset.save() # must save after editsThen we can add the metadata for each “group” of images or media files in just one place.
Learn more about Grouped datasets and Multiple media fields in the FiftyOne Docs.
Display bounding boxes without masks
Community Slack member Lukas asked,
“Is there an easy way to display only the bounding boxes and not the masks for detections that have both?”
Yes! This is possible in the Python API, where you can create a view that explicitly omits the masks while retaining the masks on the dataset:
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
session = fo.Session()
session.view = dataset.set_field("ground_truth.detections.mask", None)Learn more about views and DatasetView in the FiftyOne Docs.
Join the FiftyOne community!
Join the thousands of engineers and data scientists already using FiftyOne to solve some of the most challenging problems in computer vision today!
- 1,100+ FiftyOne Slack members
- 2,100+ stars on GitHub
- 1,900+ Meetup members
- Used by 205+ repositories
- 49+ contributors
What’s next?
- If you like what you see on GitHub, give the project a star.
- Get started! We’ve made it easy to get up and running in a few minutes.
- Join the FiftyOne Slack community, we’re always happy to help.
