Dataset Views

FiftyOne provides methods that allow you to sort, slice, and search your Dataset using any information that you have added to the Dataset. Performing these actions returns a DatasetView into your Dataset that will that will show only the samples and labels therein that match your criteria.

Overview

A DatasetView is returned whenever any sorting, slicing, or searching operation is performed on a Dataset.

You can explicitly create a view that contains an entire dataset via Dataset.view():

import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

view = dataset.view()

print(view)

Dataset:        quickstart
Media type:     image
Num samples:    200
Tags:           ['validation']
Sample fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    uniqueness:   fiftyone.core.fields.FloatField
    predictions:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    ---

You can access specific information about a view in the natural ways:

len(view)
# 200

view.media_type
# "image"

Note

DatasetView does not hold its contents in-memory; it contains a pipeline of operations that define what samples will be loaded when the contents of the view are accessed.

Like datasets, you access the samples in a view by iterating over it:

for sample in view:
    # Do something with `sample`

Or, you can access individual samples in a view by their ID or filepath:

sample = view.take(1).first()

print(type(sample))
# fiftyone.core.sample.SampleView

same_sample = view[sample.id]
also_same_sample = view[sample.filepath]

view[other_sample_id]
# KeyError: sample non-existent or not in view

Note

Accessing samples in a DatasetView returns SampleView objects, not Sample objects. The two classes are largely interchangable, but SampleView provides some extra features. See filtering sample contents for more details.

View stages

Dataset views encapsulate a pipeline of logical operations that determine which samples appear in the view (and perhaps what subset of their contents).

Each view operation is captured by a ViewStage:

# List available view operations on a dataset
print(dataset.list_view_stages())
# ['exclude', 'exclude_fields', 'exists', ..., 'skip', 'sort_by', 'take']

These operations are conveniently exposed as methods on Dataset instances, in which case they create an initial DatasetView:

# Random set of 100 samples from the dataset
random_view = dataset.take(100)

len(random_view)
# 100

They are also exposed on DatasetView instances, in which case they return another DatasetView with the operation appended to its internal pipeline so that multiple operations can be chained together:

# Sort `random_view` by filepath
sorted_random_view = random_view.sort_by("filepath")

The sections below discuss some interesting view stages in more detail. You can also refer to the fiftyone.core.stages module documentation for examples of using each stage.

Sorting

You can use sort_by() to sort the samples in a Dataset or DatasetView by a field of interest. The samples in the returned DatasetView can be sorted in ascending or descending order:

view = dataset.sort_by("filepath")
view = dataset.sort_by("filepath", reverse=True)

You can also sort by expressions!

from fiftyone import ViewField as F

# Sort by number of detections in `Detections` field `ground_truth`
view = dataset.sort_by(F("ground_truth.detections").length(), reverse=True)

print(len(view.first().ground_truth.detections))  # 39
print(len(view.last().ground_truth.detections))  # 0

Slicing

You can extract a range of Sample instances from a Dataset using skip() and limit() or, equivalently, by using array slicing:

# Skip the first 2 samples and take the next 3
range_view1 = dataset.skip(2).limit(3)

# Equivalently, using array slicing
range_view2 = dataset[2:5]

Samples can be accessed from views in all the same ways as for datasets. This includes using first() and last() to retrieve the first and last samples in a view, respectively, or accessing a sample directly from a DatasetView by its ID or filepath.

Note

Accessing a sample by its integer index in a DatasetView is not allowed. The best practice is to lookup individual samples by ID or filepath, or use array slicing to extract a range of samples, and iterate over samples in a view.

view[0]
# KeyError: "Accessing samples by numeric index is not supported.
# Use sample IDs, filepaths, or slices"

Shuffling

The samples in a Dataset or DatasetView can be randomly shuffled using shuffle():

# Randomly shuffle the order of the samples in the dataset
view1 = dataset.shuffle()

An optional seed can be provided to make the shuffle deterministic:

# Randomly shuffle the samples in the dataset with a fixed seed

view2 = dataset.shuffle(seed=51)
print(view2.first().id)
# 5f31bbfcd0d78c13abe159b1

also_view2 = dataset.shuffle(seed=51)
print(also_view2.first().id)
# 5f31bbfcd0d78c13abe159b1

Random sampling

You can extract a random subset of the samples in a Dataset or DatasetView using take():

# Take 5 random samples from the dataset
view1 = dataset.take(5)

An optional seed can be provided to make the sampling deterministic:

# Take 5 random samples from the dataset with a fixed seed

view2 = dataset.take(5, seed=51)
print(view2.first().id)
# 5f31bbfcd0d78c13abe159b1

also_view2 = dataset.take(5, seed=51)
print(also_view2.first().id)
# 5f31bbfcd0d78c13abe159b1

Filtering

The real power of DatasetView is the ability to write your own search queries based on your data.

Querying samples

You can query for a subset of the samples in a dataset via the match() method. The syntax is:

match_view = dataset.match(expression)

where expression defines the matching expression to use to decide whether to include a sample in the view.

FiftyOne provides powerful ViewField and ViewExpression classes that allow you to use native Python operators to define your match expression. Simply wrap the target field of your sample in a ViewField and then apply comparison, logic, arithmetic or array operations to it to create a ViewExpression. You can use dot notation to refer to fields or subfields of the embedded documents in your samples. Any resulting ViewExpression that returns a boolean is a valid expression!

The code below shows a few examples. See the API reference for ViewExpression for a full list of supported operations.

from fiftyone import ViewField as F

# Populate metadata on all samples
dataset.compute_metadata()

# Samples whose image is less than 48 KB
small_images_view = dataset.match(F("metadata.size_bytes") < 48 * 1024)

# Samples that contain at least one prediction with confidence above 0.99
# or whose label ifs "cat" or "dog"
match = (F("confidence") > 0.99) | (F("label").is_in(("cat", "dog")))
matching_view = dataset.match(
    F("predictions.detections").filter(match).length() > 0
)

Common filters

Convenience functions for common queries are also available.

Use the match_tags() method to match samples that have the specified tag(s) in their tags field:

# The validation split of the dataset
val_view = dataset.match_tags("validation")

# Union of the validation and test splits
val_test_view = dataset.match_tags(("validation", "test"))

Use exists() to only include samples for which a given Field exists and is not None:

# The subset of samples where predictions have been computed
predictions_view = dataset.exists("predictions")

Use select() and exclude() to restrict attention to or exclude samples from a view by their IDs:

# Get the IDs of two random samples
sample_ids = [
    dataset.take(1).first().id,
    dataset.take(1).first().id,
]

# Include only samples with the given IDs in the view
selected_view = dataset.select(sample_ids)

# Exclude samples with the given IDs from the view
excluded_view = dataset.exclude(sample_ids)

Filtering sample contents

Dataset views can also be used to filter the contents of samples in the view. That’s why DatasetView instances return SampleView objects rather than Sample objects.

SampleView instances represent the content of your samples in all of the usual ways, with some important caveats:

• If you modify the contents of a SampleView and then save() it, any changes that you made to the contents of the SampleView will be reflected in the database.

• Sample views can exclude fields and filter elements of a field (e.g., omit certain detections from an array of detections in the sample). This means that SampleView instances need not contain all of the information in a sample.

• Sample views are not singletons and thus you must explicitly reload() them in order to refresh their contents if the underlying sample has been modified elsewhere. However, extracting a SampleView from a DatasetView always returns the updated version of the sample’s contents.

You can use the select_fields() and exclude_fields() stages to select or exclude fields from the returned SampleView:

for sample in dataset.select_fields("ground_truth"):
    print(sample.id)            # OKAY: `id` is always available
    print(sample.ground_truth)  # OKAY: `ground_truth` was selected
    print(sample.predictions)   # AttributeError: `predictions` was not selected

for sample in dataset.exclude_fields("predictions"):
    print(sample.id)            # OKAY: `id` is always available
    print(sample.ground_truth)  # OKAY: `ground_truth` was not excluded
    print(sample.predictions)   # AttributeError: `predictions` was excluded

The filter_labels() stage is a powerful stage that allows you to filter the contents of Detections, Classifications, Polylines, and Keypoints fields, respectively.

Here are some self-contained examples for each task:

Classifications

Detections

Polylines

Keypoints

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("imagenet-sample")

# Only include samples whose ground truth `label` is "slug" or "conch"
slug_conch_view = dataset.filter_labels(
    "ground_truth", (F("label") == "slug") | (F("label") == "conch")
)

session = fo.launch_app(view=slug_conch_view)

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Bboxes are in [top-left-x, top-left-y, width, height] format
bbox_area = F("bounding_box")[2] * F("bounding_box")[3]

# Only includes predictions whose bounding boxes have an area of at
# least 50% of the image, and only include samples with at least
# one prediction after filtering
large_boxes_view = dataset.filter_labels("predictions", bbox_area >= 0.5)

session = fo.launch_app(view=large_boxes_view)

Note

See the BDD100K dataset in the Dataset Zoo for download instructions.

import fiftyone as fo
import fiftyone.zoo as foz

# The path to the source files that you manually downloaded
source_dir = "/path/to/dir-with-bdd100k-files"

dataset = foz.load_zoo_dataset(
    "bdd100k", split="validation", source_dir=source_dir
)

# Only include polylines that are filled (polygons, not polylines),
# and only include samples with at least one polygon after filtering
polygons_view = dataset.filter_labels("gt_polylines", F("filled") == True)

session = fo.launch_app(view=polygons_view)

Note

This example uses a Keypoint R-CNN model from the Model Zoo.

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Load a keypoint model
model = foz.load_zoo_model("keypoint-rcnn-resnet50-fpn-coco-torch")

# Grab a few samples that have people in them
person_view  = dataset.match(
    F("ground_truth.detections").map(F("label") == "person").length() > 0
).take(4)

person_view.apply_model(model, label_field="rcnn")

# Only include keypoints in the `rcnn_keypoints` field of each
# sample that have at least 10 vertices, and only include samples
# with at least one keypoint instance after filtering
many_points_view = dataset.filter_labels(
    "rcnn_keypoints", F("points").length() >= 10,
)

session = fo.launch_app(view=many_points_view)

You can also use the filter_field() stage to filter the contents of arbitrarily-typed fields:

# Remove tags from samples that don't include the "validation" tag
clean_tags_view = dataset.filter_field("tags", F().contains("validation"))

Note

When you create a DatasetView that contains filtered detections or classifications, the other labels are not removed from the source dataset, even if you save() a SampleView after modifying the filtered detections. This is becauase each label is updated individually, and other labels in the field are left unchanged.

view = dataset.filter_labels("predictions", ...)

for sample in view:
    predictions = sample.predictions

    # Modify the detections in the view
    for detection in predictions.detections:
        detection["new_field"] = True

    # Other detections in the `predictions` field of the samples that
    # did not appear in the `view` are not deleted or modified
    sample.save()

If you do want to delete data from your samples, assign a new value to the field:

view = dataset.filter_labels("predictions", ...)

for sample in view:
    sample.predictions = fo.Detections(...)

    # Existing detections in the `predictions` field of the samples
    # are deleted
    sample.save()

Date-based views

If your dataset contains date fields, you can construct dataset views that query/filter based on this information by simply writing the appropriate ViewExpression, using date, datetime and timedelta objects to define the required logic.

For example, you can use the match() stage to filter a dataset by date as follows:

from datetime import datetime, timedelta

import fiftyone as fo
from fiftyone import ViewField as F

dataset = fo.Dataset()
dataset.add_samples(
    [
        fo.Sample(
            filepath="image1.png",
            capture_date=datetime(2021, 8, 24, 1, 0, 0),
        ),
        fo.Sample(
            filepath="image2.png",
            capture_date=datetime(2021, 8, 24, 2, 0, 0),
        ),
        fo.Sample(
            filepath="image3.png",
            capture_date=datetime(2021, 8, 24, 3, 0, 0),
        ),
    ]
)

query_date = datetime(2021, 8, 24, 2, 1, 0)
query_delta = timedelta(minutes=30)

# Samples with capture date after 2021-08-24 02:01:00
view = dataset.match(F("capture_date") > query_date)
print(view)

# Samples with capture date within 30 minutes of 2021-08-24 02:01:00
view = dataset.match(abs(F("capture_date") - query_date) < query_delta)
print(view)

Note

As the example above demonstrates, ViewExpression instances may contain date, datetime and timedelta objects. Internally, subtracting two dates returns the number of milliseconds between them. Using timedelta allows these units to be abstracted away from the user.

Object patches

If your dataset contains label list fields like Detections or Polylines, then you can use to_patches() to create views that contain one sample per object patch in a specified label field of your dataset.

For example, you can extract patches for all ground truth objects in a detection dataset:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

session = fo.launch_app(dataset)

# Convert to ground truth patches
gt_patches = dataset.to_patches("ground_truth")
print(gt_patches)

# View patches in the App
session.view = gt_patches

Dataset:     quickstart
Media type:  image
Num patches: 1232
Tags:        ['validation']
Patch fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    sample_id:    fiftyone.core.fields.ObjectIdField
    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detection)
View stages:
    1. ToPatches(field='ground_truth', config=None)

Note

You can pass the optional other_fields pararmeter to to_patches() to specify additional read-only sample-level fields that each patch should include from their parent samples.

Or, you could chain view stages to create a view that contains patches for a filtered set of predictions:

# Now extract patches for confident person predictions
person_patches = (
    dataset
    .filter_labels(
        "predictions",
        (F("label") == "person") & (F("confidence") > 0.9)
    )
    .to_patches("predictions")
)
print(person_patches)

# View patches in the App
session.view = person_patches

Note

Did you know? You can convert to object patches view directly from the App!

Object patches views are just like any other dataset view in the sense that:

• You can append view stages via the App view bar or views API

• Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

• Any modifications to the Label elements in the patches view that you make by iterating over the contents of the view or calling set_values() will be reflected on the source dataset

• Calling save() on an object patches view (typically one that contains additional view stages that filter or modify its contents) will sync any Label edits or deletions with the source dataset

However, because object patches views only contain a subset of the contents of a Sample from the source dataset, there are some differences compared to non-patch views:

• Tagging or untagging patches (as opposed to their labels) will not affect the tags of the underlying Sample

• Any edits that you make to sample-level fields of object patches views other than the field that defines the patches themselves will not be reflected on the source dataset

Note

Did you know? You can export object patches as classification datasets!

Evaluation patches

If you have run evaluation on predictions from an object detection model, then you can use to_evaluation_patches() to transform the dataset (or a view into it) into a new view that contains one sample for each true positive, false positive, and false negative example.

True positive examples will result in samples with both their ground truth and predicted fields populated, while false positive/negative examples will only have one of their corresponding predicted/ground truth fields populated, respectively.

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Evaluate `predictions` w.r.t. labels in `ground_truth` field
dataset.evaluate_detections(
    "predictions", gt_field="ground_truth", eval_key="eval"
)

session = fo.launch_app(dataset)

# Convert to evaluation patches
eval_patches = dataset.to_evaluation_patches("eval")
print(eval_patches)

print(eval_patches.count_values("type"))
# {'fn': 246, 'fp': 4131, 'tp': 986}

# View patches in the App
session.view = eval_patches

Dataset:     quickstart
Media type:  image
Num patches: 5363
Tags:        ['validation']
Patch fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    predictions:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    sample_id:    fiftyone.core.fields.ObjectIdField
    type:         fiftyone.core.fields.StringField
    iou:          fiftyone.core.fields.FloatField
    crowd:        fiftyone.core.fields.BooleanField
View stages:
    1. ToEvaluationPatches(eval_key='eval', config=None)

Note

You can pass the optional other_fields pararmeter to to_patches() to specify additional read-only sample-level fields that each patch should include from their parent samples.

Refer to the evaluation guide guide for more information about running evaluations and using evaluation patches views to analyze object detection models.

Note

Did you know? You can convert to evaluation patches view directly from the App!

Evaluation patches views are just like any other dataset view in the sense that:

• You can append view stages via the App view bar or views API

• Any modifications to ground truth or predicted label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

• Any modifications to the predicted or ground truth Label elements in the patches view that you make by iterating over the contents of the view or calling set_values() will be reflected on the source dataset

• Calling save() on an evaluation patches view (typically one that contains additional view stages that filter or modify its contents) will sync any Label edits or deletions with the source dataset

However, because evaluation patches views only contain a subset of the contents of a Sample from the source dataset, there are some differences compared to non-patch views:

• Tagging or untagging patches themselves (as opposed to their labels) will not affect the tags of the underlying Sample

• Any edits that you make to sample-level fields of evaluation patches views other than the ground truth/predicted label fields will not be reflected on the source dataset

Video views

Most view stages naturally support video datasets. For example, stages that refer to fields can be applied to the frame-level fields of video samples by prepending "frames." to the relevent parameters:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that only contains vehicles
view = dataset.filter_labels("frames.detections", F("label") == "vehicle")

# Compare the number of objects in the view and the source dataset
print(dataset.count("frames.detections.detections"))  # 11345
print(view.count("frames.detections.detections"))  # 7511

In addition, FiftyOne provides a variety of dedicated view stages for performing manipulations that are unique to video data.

Clip views

You can use to_clips() to create views into your video datasets that contain one sample per clip defined by a specific field or expression in a video collection.

For example, if you have temporal detection labels on your dataset, then you can create a clips view that contains one sample per temporal segment by simply passing the name of the temporal detection field to to_clips():

import fiftyone as fo

dataset = fo.Dataset()

sample1 = fo.Sample(
    filepath="video1.mp4",
    events=fo.TemporalDetections(
        detections=[
            fo.TemporalDetection(label="meeting", support=[1, 3]),
            fo.TemporalDetection(label="party", support=[2, 4]),
        ]
    ),
)

sample2 = fo.Sample(
    filepath="video2.mp4",
    metadata=fo.VideoMetadata(total_frame_count=5),
    events=fo.TemporalDetections(
        detections=[
            fo.TemporalDetection(label="party", support=[1, 3]),
            fo.TemporalDetection(label="meeting", support=[3, 5]),
        ]
    ),
)

dataset.add_samples([sample1, sample2])

# Create a clips view with one clip per event
view = dataset.to_clips("events")
print(view)

# Verify that one sample per clip was created
print(dataset.count("events.detections"))  # 4
print(len(view))  # 4

Dataset:    2021.09.03.09.44.57
Media type: video
Num clips:  4
Tags:       []
Clip fields:
    id:        fiftyone.core.fields.ObjectIdField
    sample_id: fiftyone.core.fields.ObjectIdField
    filepath:  fiftyone.core.fields.StringField
    support:   fiftyone.core.fields.FrameSupportField
    tags:      fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    events:    fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)
Frame fields:
    id:           fiftyone.core.fields.ObjectIdField
    frame_number: fiftyone.core.fields.FrameNumberField
View stages:
    1. ToClips(field_or_expr='events', config=None)

All clips views contain a top-level support field that contains the [first, last] frame range of the clip within filepath, which points to the source video.

Note that the events field, which had type TemporalDetections in the source dataset, now has type Classification in the clips view, since each classification has a one-to-one relationship with its clip.

Note

You can pass the optional other_fields pararmeter to to_clips() to specify additional read-only sample-level fields that each clip should include from their parent samples.

Note

If you edit the support or Classification of a sample in a clips view created from temporal detections, the changes will be applied to the corresponding TemporalDetection in the source dataset.

Continuing from the example above, if you would like to see clips only for specific temporal detection labels, you can achieve this by first filtering the labels:

from fiftyone import ViewField as F

# Create a clips view with one clip per meeting
view = (
    dataset
    .filter_labels("events", F("label") == "meeting")
    .to_clips("events")
)

print(view.values("events.label"))
# ['meeting', 'meeting']

Clips views can also be created based on frame-level labels, which provides a powerful query language that you can use to find segments of a video dataset that contain specific frame content of interest.

In the simplest case, you can provide the name of a frame-level list field (e.g., Classifications or Detections) to to_clips(), which will create one clip per contiguous range of frames that contain at least one label in the specified field:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that contains one clip per contiguous range of frames that
# contains at least one detection
view = dataset.to_clips("frames.detections")
print(view)

The above view turns out to not be very interesting, since every frame in the quickstart-video dataset contains at least one object. So, instead, lets first filter the objects so that we can construct a clips view that contains one clip per contiguous range of frames that contains at least one person:

from fiftyone import ViewField as F

# Create a view that contains one clip per contiguous range of frames that
# contains at least one person
view = (
    dataset
    .filter_labels("frames.detections", F("label") == "person")
    .to_clips("frames.detections")
)
print(view)

Dataset:    quickstart-video
Media type: video
Num clips:  8
Tags:       []
Clip fields:
    id:        fiftyone.core.fields.ObjectIdField
    sample_id: fiftyone.core.fields.ObjectIdField
    filepath:  fiftyone.core.fields.StringField
    support:   fiftyone.core.fields.FrameSupportField
    tags:      fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:  fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
Frame fields:
    id:           fiftyone.core.fields.ObjectIdField
    frame_number: fiftyone.core.fields.FrameNumberField
    detections:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    1. FilterLabels(field='frames.detections', filter={'$eq': ['$$this.label', 'person']}, only_matches=True)
    2. ToClips(field_or_expr='frames.detections', config=None)

When you iterate over the frames of a sample in a clip view, you will only get the frames within the [first, last] support of each clip:

sample = view.last()

print(sample.support)
# [116, 120]

frame_numbers = []
for frame_number, frame in sample.frames.items():
    frame_numbers.append(frame_number)

print(frame_numbers)
# [116, 117, 118, 119, 120]

Note

Clips views created via to_clips() always contain all frame-level labels from the underlying dataset for their respective frame supports, even if frame-level filtering was applied in previous view stages. In other words, filtering prior to the to_clips() stage only affects the frame supports.

You can, however, apply frame-level filtering to clips by appending filtering operations after the to_clips() stage in your view, just like any other view.

More generally, you can provide an arbitrary ViewExpression to to_clips() that defines a boolean expression to apply to each frame. In this case, the clips view will contain one clip per contiguous range of frames for which the expression evaluates to true:

# Create a view that contains one clip per contiguous range of frames that
# contains at least 10 vehicles
view = (
    dataset
    .filter_labels("frames.detections", F("label") == "vehicle")
    .to_clips(F("detections.detections").length() >= 10)
)
print(view)

See this section for more information about constructing frame expressions.

Note

You can pass optional tol and min_len pararmeters to to_clips() to configure a missing frame tolerance and minimum length for clips generated from frame-level fields or expressions.

Clip views are just like any other dataset view in the sense that:

• You can append view stages via the App view bar or views API

• Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

• Any modifications to the frame-level labels in a clips view that you make by iterating over the contents of the view or calling set_values() will be reflected on the source dataset

• Calling save() on a clips view (typically one that contains additional view stages that filter or modify its contents) will sync any frame-level edits or deletions with the source dataset

However, because clip views represent only a subset of a Sample from the source dataset, there are some differences compared to non-clip views:

• Tagging or untagging clips (as opposed to their labels) will not affect the tags of the underlying Sample

• Any edits that you make to sample-level fields of clip views will not be reflected on the source dataset (except for edits to the support and Classification field populated when generating clip views based on TemporalDetection labels, as described above)

Frame views

Use to_frames() to create image views into your video datasets that contain one sample per video frame in the dataset.

Note

Did you know? Using to_frames() enables you to execute workflows such as model evaluation and Brain methods that only support image collections to the frames of your video datasets!

In the simplest case, you can create a view that contains a sample for every frame of the videos in a Dataset or DatasetView:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

session = fo.launch_app(dataset)

# Create a frames view for the entire dataset
frames = dataset.to_frames()
print(frames)

# Verify that one sample per frame was created
print(dataset.sum("metadata.total_frame_count"))  # 1279
print(len(frames))  # 1279

# View frames in the App
session.view = frames

Dataset:     quickstart-video
Media type:  image
Num samples: 1279
Tags:        []
Sample fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    sample_id:    fiftyone.core.fields.ObjectIdField
    frame_number: fiftyone.core.fields.FrameNumberField
    detections:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    1. ToFrames(config=None)

Note

Unless you have configured otherwise, to_frames() will sample the necessary frames from the input video collection into directories of per-frame images when the view is created. For large video datasets, this may take some time and require substantial disk space.

Frames that have previously been sampled will not be resampled, so creating frame views into the same dataset will become faster after the frames have been sampled.

More generally, to_frames() exposes a variety of parameters that you can use to configure the behavior of the video-to-image conversion process.

For example, the snippet below creates a frames view that only contains samples for frames with at least 10 objects, sampling at most one frame per second:

from fiftyone import ViewField as F

#
# Create a frames view that only contains frames with at least 10
# objects, sampled at a maximum frame rate of 1fps
#

num_objects = F("detections.detections").length()
view = dataset.match_frames(num_objects > 10)

frames = view.to_frames(max_fps=1, sparse=True)
print(frames)

# Compare the number of frames in each step
print(dataset.count("frames"))  # 1279
print(view.count("frames"))  # 354
print(len(frames))  # 13

# View frames in the App
session.view = frames

Frame views inherit all frame-level labels from the source video dataset, including their frame number. Each frame sample is also given a sample_id field that records the ID of the parent video sample, and any tags of the parent video sample are also included.

Frame views are just like any other image collection view in the sense that:

• You can append view stages via the App view bar or views API

• Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

• Any edits (including additions, modifications, and deletions) to the fields of the samples in a frames view that you make by iterating over the contents of the view or calling set_values() will be reflected on the source dataset

• Calling save() on a frames view (typically one that contains additional view stages that filter or modify its contents) will sync any changes to the frames of the underlying video dataset

The only way in which frames views differ from regular image collections is that changes to the tags or metadata fields of frame samples will not be propagated to the frames of the underlying video dataset.

Frame patches views

Since frame views into video datasets behave just like any other view, you can chain to_frames() and to_patches() to create frame patch views into your video datasets that contain one sample per object patch in the frames of the dataset!

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

session = fo.launch_app(dataset)

# Create a frame patches view
frame_patches = dataset.to_frames().to_patches("detections")
print(frame_patches)

# Verify that one sample per object was created
print(dataset.count("frames.detections.detections"))  # 11345
print(len(frame_patches))  # 11345

# View frame patches in the App
session.view = frame_patches

Dataset:     quickstart-video
Media type:  image
Num patches: 11345
Tags:        []
Patch fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    sample_id:    fiftyone.core.fields.ObjectIdField
    frame_id:     fiftyone.core.fields.ObjectIdField
    frame_number: fiftyone.core.fields.FrameNumberField
    detections:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detection)
View stages:
    1. ToFrames(config=None)
    2. ToPatches(field='detections', config=None)

Querying frames

You can query for a subset of the frames in a video dataset via match_frames(). The syntax is:

match_view = dataset.match_frames(expression)

where expression defines the matching expression to use to decide whether to include a frame in the view.

FiftyOne provides powerful ViewField and ViewExpression classes that allow you to use native Python operators to define your match expression. Simply wrap the target frame field in a ViewField and then apply comparison, logic, arithmetic or array operations to it to create a ViewExpression. You can use dot notation to refer to fields or subfields of the embedded documents in your frames. Any resulting ViewExpression that returns a boolean is a valid expression!

The snippet below demonstrates a possible workflow. See the API reference for ViewExpression for a full list of supported operations.

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that only contains frames with at least 10 objects
num_objects = F("detections.detections").length()
view = dataset.match_frames(num_objects > 10)

# Compare the number of frames in each collection
print(dataset.count("frames"))  # 1279
print(view.count("frames"))  # 354

You can also use select_frames() and exclude_frames() to restrict attention to or exclude frames from a view by their IDs:

# Get the IDs of a couple frames
frame_ids = [
    dataset.first().frames.first().id,
    dataset.last().frames.last().id,
]

# Select only the specified frames
selected_view = dataset.select_frames(frame_ids)

# Exclude frames with the given IDs from the view
excluded_view = dataset.exclude_frames(frame_ids)

Visual similarity

If you have indexed your dataset by visual similarity, then you can use the sort_by_similarity() stage to programmatically query your data by visual similarity to image(s) or object patch(es) of interest.

Image similarity

The example below indexes a dataset by image similarity using compute_similarity() and then uses sort_by_similarity() to sort the dataset by visual similarity to a chosen image:

import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index the dataset by image similarity
fob.compute_similarity(dataset, brain_key="image_sim")

session = fo.launch_app(dataset)

# Select a random query image
query_id = dataset.take(1).first().id

# Sort the samples by visual similarity to the query image
view = dataset.sort_by_similarity(query_id, brain_key="image_sim")
print(view)

# View results in the App
session.view = view

Note

Refer to the Brain guide for more information about generating similarity indexes, and check out the App guide to see how to sort images by visual similarity via point-and-click in the App!

Object similarity

The example below indexes the objects in a Detections field of a dataset by similarity using compute_similarity() and then uses sort_by_similarity() to retrieve the 15 most visually similar objects to a chosen object:

import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index the dataset by `ground_truth` object similarity
fob.compute_similarity(
    dataset, patches_field="ground_truth", brain_key="gt_sim"
)

# Convert to ground truth patches view
patches = dataset.to_patches("ground_truth")

# View patches in the App
session = fo.launch_app(view=patches)

# Select a random query object
query_id = patches.take(1).first().id

# Retrieve the 15 most visually similar objects
similar_objects = patches.sort_by_similarity(query_id, k=15, brain_key="gt_sim")

# View results in the App
session.view = similar_objects

Note

Refer to the Brain guide for more information about generating similarity indexes, and check out the App guide to see how to sort objects by visual similarity via point-and-click in the App!

Geolocation

If your samples have geolocation data, then you can use the geo_near() and geo_within() stages to filter your data based on their location.

For example, you can use geo_near() to sort your samples by proximity to a location:

import fiftyone as fo
import fiftyone.zoo as foz

TIMES_SQUARE = [-73.9855, 40.7580]

dataset = foz.load_zoo_dataset("quickstart-geo")

# Sort the samples by their proximity to Times Square, and only include
# samples within 5km
view = dataset.geo_near(TIMES_SQUARE, max_distance=5000)

Or, you can use geo_within() to only include samples that lie within a longitude-latitude polygon of your choice:

import fiftyone as fo
import fiftyone.zoo as foz

MANHATTAN = [
    [
        [-73.949701, 40.834487],
        [-73.896611, 40.815076],
        [-73.998083, 40.696534],
        [-74.031751, 40.715273],
        [-73.949701, 40.834487],
    ]
]

dataset = foz.load_zoo_dataset("quickstart-geo")

# Only contains samples in Manhattan
view = dataset.geo_within(MANHATTAN)

Tagging contents

You can use the tag_samples() and untag_samples() methods to add or remove sample tags from the samples in a view:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
dataset.untag_samples("validation") # remove pre-existing tags

# Perform a random 90-10 test-train split
dataset.take(0.1 * len(dataset)).tag_samples("test")
dataset.match_tags("test", bool=False).tag_samples("train")

print(dataset.count_sample_tags())
# {'train': 180, 'test': 20}

You can also use the tag_labels() and untag_labels() methods to add or remove label tags from the labels in one or more fields of a view:

# Add a tag to all low confidence predictions
view = dataset.filter_labels("predictions", F("confidence") < 0.06)
view.tag_labels("low_confidence", label_fields="predictions")

print(dataset.count_label_tags())
# {'low_confidence': 447}

Editing fields

You can perform arbitrary edits to a DatasetView by iterating over its contents and editing the samples directly:

import random

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

view = dataset.limit(50)

# Populate a new field on each sample in the view
for sample in view:
    sample["random"] = random.random()
    sample.save()

print(dataset.count("random"))  # 50
print(dataset.bounds("random")) # (0.0005, 0.9928)

Alternatively, you can use set_values() to set a field (or embedded field) on each sample in the collection in a single batch operation:

# Delete the field we added in the previous variation
dataset.delete_sample_field("random")

# Equivalent way to populate a new field on each sample in a view
values = [random.random() for _ in range(len(view))]
view.set_values("random", values)

print(dataset.count("random"))  # 50
print(dataset.bounds("random")) # (0.0272, 0.9921)

Note

When possible, using set_values() is often more efficient than performing the equivalent operation via an explicit iteration over the DatasetView because it avoids the need to read the entire SampleView instances into memory and then save them.

Naturally, you can edit nested sample fields of a DatasetView by iterating over the view and editing the necessary data:

# Create a view that contains only low confidence predictions
view = dataset.filter_labels("predictions", F("confidence") < 0.06)

# Add a tag to all predictions in the view
for sample in view:
    for detection in sample["predictions"].detections:
        detection.tags.append("low_confidence")

    sample.save()

print(dataset.count_label_tags())
# {'low_confidence': 447}

However, an equivalent and often more efficient approach is to use values() to extract the slice of data you wish to modify and then use set_values() to save the updated data in a single batch operation:

# Remove the tags we added in the previous variation
dataset.untag_labels("low_confidence")

# Load all predicted detections
# This is a list of lists of `Detection` instances for each sample
detections = view.values("predictions.detections")

# Add a tag to all low confidence detections
for sample_detections in detections:
    for detection in sample_detections:
        detection.tags.append("low_confidence")

# Save the updated predictions
view.set_values("predictions.detections", detections)

print(dataset.count_label_tags())
# {'low_confidence': 447}

Transforming fields

In certain situations, you may wish to temporarily modify the values of sample fields in the context of a DatasetView without modifying the underlying dataset. FiftyOne provides the set_field() and map_labels() methods for this purpose.

For example, suppose you would like to rename a group of labels to a single category in order to run your evaluation routine. You can use map_labels() to do this:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

ANIMALS = [
    "bear", "bird", "cat", "cow", "dog", "elephant", "giraffe",
    "horse", "sheep", "zebra"
]

# Replace all animal detection's labels with "animal"
mapping = {k: "animal" for k in ANIMALS}
animals_view = dataset.map_labels("predictions", mapping)

counts = animals_view.count_values("predictions.detections.label")
print(counts["animal"])
# 529

Or, suppose you would like to lower bound all confidences of objects in the predictions field of a dataset. You can use set_field() to do this:

# Lower bound all confidences in the `predictions` field to 0.5
bounded_view = dataset.set_field(
    "predictions.detections.confidence",
    F("confidence").max(0.5),
)

print(bounded_view.bounds("predictions.detections.confidence"))
# (0.5, 0.9999035596847534)

Note

In order to populate a new field using set_field(), you must first declare the new field on the dataset via add_sample_field():

# Record the number of predictions in each sample in a new field
dataset.add_sample_field("num_predictions", fo.IntField)
view = dataset.set_field("num_predictions", F("predictions.detections").length())

view.save("num_predictions")  # save the new field's values on the dataset
print(dataset.bounds("num_predictions"))  # (1, 100)

The ViewExpression language is quite powerful, allowing you to define complex operations without needing to write an explicit Python loop to perform the desired manipulation.

For example, the snippet below visualizes the top-5 highest confidence predictions for each sample in the dataset:

from fiftyone import ViewField as F

# Extracts the 5 highest confidence predictions for each sample
top5_preds = F("detections").sort_by("confidence", reverse=True)[:5]

top5_view = (
    dataset
    .set_field("predictions.detections", top5_preds)
    .select_fields("predictions")
)

session = fo.launch_app(view=top5_view)

If you want to permanently save transformed view fields to the underlying dataset, you can do so by calling save() on the view and optionally passing the name(s) of specific field(s) that you want to save:

# Saves `predictions` field's contents in the view permanently to dataset
top5_view.save("predictions")

Saving and cloning

Ordinarily, when you define a DatasetView that extracts a specific subset of a dataset and its fields, the underlying Dataset is not modified. However, you can use save() to overwrite the underlying dataset with the contents of a view you’ve created:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

# Discard all predictions with confidence below 0.3
high_conf_view = dataset.filter_labels("predictions", F("confidence") > 0.3)
high_conf_view.save()

print(dataset.bounds("predictions.detections.confidence"))
# (0.3001, 0.9999)

Alternatively, you can create a new Dataset that contains only the contents of a DatasetView using clone():

# Reload full quickstart dataset
dataset.delete()
dataset = foz.load_zoo_dataset("quickstart")

# Create a new dataset that contains only the high confidence predictions
high_conf_view = dataset.filter_labels("predictions", F("confidence") > 0.3)
high_conf_dataset = high_conf_view.clone()

print(high_conf_dataset.bounds("predictions.detections.confidence"))
# (0.3001, 0.9999)

You can also use clone_sample_field() to copy the contents of a view’s field into a new field of the underlying Dataset:

print(dataset.count("predictions.detections"))  # 5620

# Make view containing only high confidence predictions
view = dataset.filter_labels("predictions", F("confidence") > 0.5)
print(view.count("predictions.detections"))  # 1564

# Copy high confidence predictions to a new field
view.clone_sample_field("predictions", "high_conf_predictions")
print(dataset.count("high_conf_predictions.detections"))  # 1564

Tips & tricks

Chaining view stages

View stages can be chained together to perform complex operations:

from fiftyone import ViewField as F

# Extract the first 5 samples with the "validation" tag, alphabetically by
# filepath, whose images are >= 48 KB
complex_view = (
    dataset
    .match_tags("validation")
    .exists("metadata")
    .match(F("metadata.size_bytes") >= 48 * 1024)  # >= 48 KB
    .sort_by("filepath")
    .limit(5)
)

Filtering detections by area

Need to filter your detections by bounding box area? Use this ViewExpression!

from fiftyone import ViewField as F

# Bboxes are in [top-left-x, top-left-y, width, height] format
bbox_area = F("bounding_box")[2] * F("bounding_box")[3]

# Only contains boxes whose area is between 5% and 50% of the image
medium_boxes_view = dataset.filter_labels(
    "predictions", (0.05 <= bbox_area) & (bbox_area < 0.5)
)

FiftyOne stores bounding box coordinates as relative values in [0, 1]. However, you can use the expression below to filter by absolute pixel area:

from fiftyone import ViewField as F

dataset.compute_metadata()

# Computes the area of each bounding box in pixels
bbox_area = (
    F("$metadata.width") * F("bounding_box")[2] *
    F("$metadata.height") * F("bounding_box")[3]
)

# Only contains boxes whose area is between 32^2 and 96^2 pixels
medium_boxes_view = dataset.filter_labels(
    "predictions", (32 ** 2 < bbox_area) & (bbox_area < 96 ** 2)
)

Removing a batch of samples from a dataset

You can easily remove a batch of samples from a Dataset by constructing a DatasetView that contains the samples, and then deleting them from the dataset as follows:

# Choose 10 samples at random
unlucky_samples = dataset.take(10)

dataset.delete_samples(unlucky_samples)

Efficiently iterating samples

If you have a dataset with larger fields, such as Classifications or Detections, it can be expensive to load entire samples into memory. If, for a particular use case, you are only interested in a subset of fields, you can use Dataset.select_fields() to load only the fields of interest.

Let’s say you have a dataset that looks like this:

Name:           open-images-v4-test
Num samples:    1000
Persistent:     True
Tags:           []
Sample fields:
    id:                       fiftyone.core.fields.ObjectIdField
    filepath:                 fiftyone.core.fields.StringField
    tags:                     fiftyone.core.fields.ListField(StringField)
    metadata:                 fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    open_images_id:           fiftyone.core.fields.StringField
    groundtruth_image_labels: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classifications)
    groundtruth_detections:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    faster_rcnn:              fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    mAP:                      fiftyone.core.fields.FloatField
    AP_per_class:             fiftyone.core.fields.DictField

and you want to get a list of open_images_id’s for all samples in the dataset. Loading other fields is unnecessary; in fact, using Dataset.select_fields() to load only the open_images_id field speeds up the operation below by ~200X!

import time

start = time.time()
oiids = [s.open_images_id for s in dataset]
print(time.time() - start)
# 38.212332010269165

start = time.time()
oiids = [s.open_images_id for s in dataset.select_fields("open_images_id")]
print(time.time() - start)
# 0.20824909210205078