Announcing FiftyOne 0.21 with Operators, Dynamic Groups, and Custom Color Schemes

Voxel51 in conjunction with the FiftyOne community is excited to announce the general availability of FiftyOne 0.21. This release is packed with upgrades to FiftyOne’s plugin framework and a variety of App features that unlock new ways to explore your datasets and customize your App experience to suit your needs. How? Read on!

Wait, what’s FiftyOne?

FiftyOne is the open source machine learning toolset that enables data science teams to improve the performance of their computer vision models by helping them curate high quality datasets, evaluate models, find mistakes, visualize embeddings, and get to production faster.

• If you like what you see on GitHub, give the project a star.
• Get started! We’ve made it easy to get up and running in a few minutes.
• Join the FiftyOne Slack community, we’re always happy to help.

Ok, let’s dive into the release.

tl;dr: What’s new in FiftyOne 0.21?

This release includes:

• Operators: we added a new concept called Operators to FiftyOne’s Plugin framework that you can use to trigger arbitrary custom functionality directly from the App via a powerful custom form language, all written in Python(!)
• Dynamic groups: you can now dynamically group the samples in your dataset (in the App or Python) based on any field or expression of interest, allowing you to both quickly navigate between groups and explore variations within a particular group
• Custom color schemes: you now have full control to customize the field and label colors that are used to render content in the App!
• Custom field visibility: there’s a new settings menu in the App sidebar that you can use to visually configure which sample fields appear in the App
• Multiple point cloud overlays: you can now overlay multiple point clouds into the same scene in the App

Check out the release notes for a full rundown of additional enhancements and bugfixes in FiftyOne 0.21.

By the way, FiftyOne Teams 1.3 is also generally available! Check out its release notes to explore the new collaboration-minded features we rolled out to commercial users (remember, Teams is fully compatible with existing FiftyOne workflows).

See a recap of the live demo and AMA held on June 15

Want to see FiftyOne 0.21 in action, along with VoxelGPT, your new AI assistant for computer vision? Check out the webinar recap video and blog post. You’ll get a taste of all the amazing capabilities you can build using plugins and operators, plus all the capabilities unlocked by VoxelGPT (which is a plugin)!

Now, here’s a quick overview of some of the new features we packed into this release.

Operators

FiftyOne 0.21 adds a powerful new feature called Operators to FiftyOne’s Plugin framework that allows you to trigger arbitrary custom functionality directly from the App via custom forms, all written in Python.

For example, wish you could rename a sample field from within the App? Now you can add it yourself!

And here are the few lines of Python code required to do it, from soup to nuts:

import fiftyone.operators as foo
import fiftyone.operators.types as types

class RenameSampleField(foo.Operator):
    @property
    def config(self):
        return foo.OperatorConfig(
            name="rename_sample_field",
            label="Rename sample field",
            dynamic=True,
        )

    def resolve_input(self, ctx):
        inputs = types.Object()
        fields = ctx.dataset.get_field_schema(flat=True)
        field_keys = list(fields.keys())
        field_selector = types.AutocompleteView()
        for key in field_keys:
            field_selector.add_choice(key, label=key)

        inputs.enum(
            "field_name",
            field_keys,
            label="Field to rename",
            view=field_selector,
            required=True,
        )
        field_name = ctx.params.get("field_name", None)
        new_field_name = ctx.params.get("new_field_name", None)
        if field_name and field_name in field_keys:
            new_field_prop = inputs.str(
                "new_field_name",
                required=True,
                label="New field name",
                default=f"{field_name}_copy",
            )
            if new_field_name and new_field_name in field_keys:
                new_field_prop.invalid = True
                new_field_prop.error_message = (
                    f"Field '{new_field_name}' already exists"
                )
                inputs.str(
                    "error",
                    label="Error",
                    view=types.Error(
                        label="Field already exists",
                        description=f"Field '{new_field_name}' already exists",
                    ),
                )

        return types.Property(
            inputs, view=types.View(label="Rename sample field")
        )

    def execute(self, ctx):
        ctx.dataset.rename_sample_field(
            ctx.params["field_name"],
            ctx.params["new_field_name"],
        )
        ctx.trigger("reload_dataset")

def register(p)
    p.register(RenameSampleField)

In the above code:

• config defines the name and label (display name) of the operator, which you can access by clicking the Browse operations button in the App as shown in the GIF above
• resolve_input() defines the form that accepts user input in the App. FiftyOne provides a rich set of prebuilt types (dropdowns, selectors, etc) that you can use to build sophisticated custom forms
• execute() defines the operation to perform when the user invokes it
• ctx is a context object that provides access to the current state of the App, including things like form inputs, the current dataset, the current view, whether any samples are currently selected, etc., that you can use to implement the desired action

Operators represent bite-sized pieces of functionality that can be executed either directly by users or, importantly, chained together for internal use to build up more complex sequences of instructions.

For example, the ctx.trigger("reload_dataset") statement above demonstrates how operators can invoke other operators. In this case a builtin reload_dataset operator is triggered that causes the dataset to be reloaded in the App to pull in the schema changes that our custom Operator performed.

How do Operators relate to Plugins? FiftyOne Plugins contain zero or more Operators, which are automatically added to the Operator browser in the App whenever you install the plugin. Refer to the docs for more information about writing your own Operators and Plugins and distributing them via GitHub.

Want to get started with Operators without writing any code? Check out the fiftyone-plugins repository on GitHub to explore a growing list of plugins that you can easily download and install via the CLI:

# Download plugin(s) from a GitHub repository
fiftyone plugins download https://github.com/<user>/<repo>[/tree/branch]

# Install any requirements required to use the plugin
fiftyone plugins requirements <plugin-name> -–install

# List the plugins that you’ve downloaded locally
fiftyone plugins list

# List the available operators that you’ll be able to run
fiftyone operators list

Dynamic groups

FiftyOne 0.21 adds a new data exploration feature we’re calling dynamic groups.

With dynamic groups you can organize the samples in your dataset by a particular field or expression. For example, you can group a classification dataset like CIFAR-10 by label:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("cifar10", split="test")

# Take 100 samples and group by ground truth label
view = dataset.take(100, seed=51).group_by("ground_truth.label")

print(view.media_type)  # group
print(len(view))  # 10

which yields a new view that contains 10 groups, one per each object class.

When you iterate over a dynamic grouped view, you get one example from each group. Like any other view, you can chain additional view stages to further refine the view’s contents:

# Sort the groups by label
sorted_view = view.sort_by("ground_truth.label")

for sample in sorted_view:
    print(sample.ground_truth.label)

airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck

In addition, you can use get_dynamic_group() to retrieve a view containing all samples within a specific group:

group = view.get_dynamic_group("horse")
print(len(group))  # 11

You can use the new group action in the App’s menu to dynamically group your samples by a field of your choice directly from within the App:

In this mode, the App’s grid shows the first sample from each group, and you can click on a sample to view all elements of the group in the modal.

By default, the samples in each group are unordered, but you can provide the optional order_by and reverse arguments to group_by() to specify an ordering for the samples in each group:

# Create an image dataset that contains one sample per frame of the
# `quickstart-video` dataset
dataset = (
    foz.load_zoo_dataset("quickstart-video")
    .to_frames(sample_frames=True)
    .clone()
)

print(len(dataset))  # 1279

# Group by video ID and order each group by frame number
view = dataset.group_by("sample_id", order_by="frame_number")

print(len(view))  # 10
print(view.values("frame_number"))
# [1, 1, 1, ..., 1]

sample_id = dataset.take(1).first().sample_id
video = view.get_dynamic_group(sample_id)

print(video.values("frame_number"))
# [1, 2, 3, ..., 120]

And as usual, you can achieve the same result directly in the App:

When viewing ordered groups in the App, the modal shows a pagination UI at the bottom that you can use to navigate sequentially or via random access through the elements of the group.

Bonus: you can even group by arbitrary expressions!

from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

# Group samples by the number of ground truth objects they contain
expr = F("ground_truth.detections").length()
view = dataset.group_by(expr)

print(len(view))  # 26
print(len(dataset.distinct(expr)))  # 26

Check out the docs for more information about creating dynamic grouped views in Python and the App.

Custom color schemes

In FiftyOne 0.21 you can now fully customize the color scheme used by the App to render content in the grid/modal!

The simplest way to get started is to click on the new color palette icon above the sample grid. The GIF below demonstrates how to:

• Configure a custom color pool from which to draw colors for otherwise unspecified fields/values
• Configure the colors assigned to specific fields in color by field mode
• Configure the colors used to render specific annotations based on their attributes in color by value mode
• Save the customized color scheme as the default for the dataset

Note that any customizations you make only apply to the current dataset. Each time you load a new dataset, the color scheme will revert to that dataset’s default color scheme (if any) or else the global default color scheme. You can press Save as default to save your current color scheme as the dataset’s default scheme.

As you’d expect from FiftyOne, you can also programmatically define your color scheme through Python!

For example, the code below configures a custom color scheme suitable for analyzing evaluation results where:

• All true positive objects are green
• All false negatives objects are blue
• All false positives objects are red

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")
dataset.evaluate_detections(
    "predictions", gt_field="ground_truth", eval_key="eval"
)

# Create a custom color scheme
fo.ColorScheme(
    fields=[
        {
            "path": "ground_truth",
            "colorByAttribute": "eval",
            "valueColors": [
                {"value": "fn", "color": "#0000ff"},  # false negatives: blue
                {"value": "tp", "color": "#00ff00"},  # true positives: green
            ]
        },
        {
            "path": "predictions",
            "colorByAttribute": "eval",
            "valueColors": [
                {"value": "fp", "color": "#ff0000"},  # false positives: red
                {"value": "tp", "color": "#00ff00"},  # true positives: green
            ]
        }
    ]
)

# Option 1: launch App with a custom color scheme
session = fo.launch_app(dataset, color_scheme=color_scheme)

# Option 2: store the color scheme as default for the dataset
dataset.app_config.color_scheme = color_scheme
dataset.save()

In the above example, you can see TP/FP/FN colors in the App by clicking on the Color palette icon and switching to color by value mode.

Check out the docs for more information about creating and saving custom App color schemes.

Custom field visibility

Do you work with datasets that contain many fields and find your App is getting cluttered? FiftyOne 0.21 is here to help!

There’s a new settings menu in the App sidebar that you can use to visually configure which sample fields appear in the App:

In the simplest case, you can use the checkboxes to manually control which fields appear in the sidebar. By default, only top-level fields are available for selection, but if you want fine-grained control you can opt to include nested fields (e.g. dynamic attributes of your label fields) in the selection list as well.

Note that, after a field visibility change is applied, a filter icon appears to the left of the settings icon in the sidebar indicating how many fields are currently excluded. You can reset your selection by clicking this icon or reopening the modal and pressing the reset button at the bottom.

What if you want to switch between multiple contexts, each containing different sets of fields? You can persist and reload different field selections by saving views!

For more advanced use cases, you can use the Filter rule tab to define a rule that is dynamically applied to the dataset’s field metadata each time the App loads to determine which fields to include in the sidebar:

Multiple point cloud overlays

Last but not least, FiftyOne 0.21 includes a heavily requested feature by users that work with grouped datasets that contain multiple point cloud slices: you can now overlay multiple point clouds in the App’s 3D Visualizer!

Community contributions

We’re excited to announce that we’ve partnered with the amazing team at Sama to make the Sama-Coco dataset available for download in the FiftyOne Dataset Zoo!

As usual, downloading all or partial subsets of Sama-Coco from the zoo is easy:

import fiftyone as fo
import fiftyone.zoo as foz

# Load 50 random samples from the validation split
dataset = foz.load_zoo_dataset(
    "sama-coco",
    split="validation",
    max_samples=50,
    shuffle=True,
)

# Or download the entire dataset
dataset = foz.load_zoo_dataset("sama-coco")

Sama-Coco is a relabeling of the COCO-2017 dataset, an industry standard benchmark dataset for large-scale object detection and segmentation. Among other improvements (read more here), masks in Sama-Coco are tighter and many crowd instances have been decomposed into their components. The project was undertaken as part of Sama’s ongoing effort to redefine data quality for the modern age and to contribute to the wider research and development efforts of the ML community.

Here’s what Jerome Pasquero, Principal Product Manager at Sama, had to say about the collaboration:

“While re-annotating Sama-Coco, we sensed that we were working on something remarkable. However, when we had the opportunity to visualize and explore it in FiftyOne, we truly grasped its potential impact. The ability to easily compare the COCO and Sama-Coco datasets at scale allowed us to see the influence the re-annotations could have on machine learning. We are excited to partner with Voxel51 and make it available for download directly from the FiftyOne App as part of their 0.21 release.”

Also, shoutout to the following community members who contributed to FiftyOne 0.21:

• Jonathan Badger contributed #2706 – Labelstudio instances support 
• Akshit Priyesh contributed #2776 – Adding support for evaluating keypoints
• Rustem Galiullin contributed #2777 – add optional dice score computation for evaluate_segmentations 
• Felix Nobis contributed #3117 – multiclass classification support for scale.com labeling 
• Colin Wong contributed #3127 – Only match .txt files when reading YOLO labels

FiftyOne community updates

The FiftyOne community continues to grow!

• 1,700+ FiftyOne Slack members
• 3,000+ stars on GitHub
• 4,000+ Meetup members
• Used by 300+ repositories
• 60+ contributors