Announcing FiftyOne 0.25 with Python Panels, Custom Dashboards, SAM 2, and Elasticsearch Integration

I’m thrilled to announce the general availability of FiftyOne 0.25, an exciting update that brings significant upgrades to the representational power and extensibility of open source FiftyOne, including:

• Python Panels: a powerful framework for building custom App panels via a simple Python interface that includes a wealth of builtin components to convey information, create tutorials, show interactive graphs, trigger operations, and generally fine-tune your FiftyOne environment to surface the insights you need to develop high quality datasets and models. 
• Custom Dashboards: the first Python Panel! The Dashboard Panel allows you to build custom no-code dashboards that display statistics of interest about a dataset and the models you’ve trained on it.
• SAM 2: Segment Anything 2 is now available in the FiftyOne Model Zoo! Reduce the need for large-scale manual labeling efforts by using the latest multimodal foundation model to automatically segment objects across your image and video datasets.
• Elasticsearch Integration: you can now connect FiftyOne to your Elasticsearch cluster to perform text and image searches on your FiftyOne datasets at scale.

At Voxel51, our goal is to accelerate and automate your visual AI development process. FiftyOne is the place where you can test and refine your models alongside the data they’re trained on, enabling you to explore, visualize, and curate higher quality datasets, iterate more quickly, integrate flexibly into the way you work, and ultimately get (and keep!) your visual AI models in production.

With this release, we’re advancing our mission by expanding our commitment to extensibility and flexibility–which our users and customers have consistently told us is an absolutely critical requirement of their AI development platform–providing even more ways to customize your FiftyOne environment to suit the unique requirements of your domain and surface key insights into the integrity of your data and models.
You can give the latest release a test drive by installing FiftyOne locally:

pip install --upgrade fiftyone

Check out the release notes for a full rundown of additional enhancements and bugfixes in FiftyOne 0.25.

Okay, now let’s dive into some highlights from the release.

Python Panels

Ever since we introduced FiftyOne’s plugin framework in 2023, one of the most popular feature requests we’ve heard has been the ability to build custom in-App panels using an intuitive Python interface like the one that’s available to build Python Operators that encapsulate reusable backend computations.

Why? We consistently hear from users and customers that they need a tool that’s connected directly to their visual data that’s extensible enough to display domain-specific insights (e.g. measures of model performance stratified across custom slices of a dataset) and encapsulate workflows or analyses that they’ve developed in-house. Well, as of FiftyOne 0.25, Python Panels are here!

Python Panels are miniature full-featured data applications that you can open in App spaces and interactively manipulate to explore your dataset and update/respond to updates from other spaces that are currently open in the App.

Panels, like Operators, can make use of the fiftyone.operators.types module, which defines a rich builtin type system that panel developers can use to implement the layout and associated events that define their panels.

Panels can trigger both Python and JS operators, either programmatically or by interactively launching a prompt that users can fill out to provide the necessary parameters for the operator’s execution. This powerful composability allows panels to define interactive workflows that guide the user through executing workflows on their data and then interactively exploring and analyzing the results of the computation.

Panels can also interact with other components of the App, such as responding to changes in (or programmatically updating) the current dataset, view, current selection, or active sample in the modal.

What can you build with Python Panels? The Dashboard Panel is the first generally available example, but the possibilities are endless, and we’ll be using the panels framework ourselves over the next few months to roll out a variety of new in-App experiences for workflows like model evaluation, identifying data quality issues, auto-tagging, and much more! 🚀

There are a few common patterns that we’ve seen pop up when developing panels, and we’ll highlight a couple of them next. If you’re interested in developing your own panels, check out the panel examples plugin to see a collection of fully-functional panels that demonstrate these common patterns in action.

Interactive Plots

Panels provide native support for defining interactive plots that can render data from the current dataset and dynamically update or trigger actions as users interact with the plots. Such interactivity brings your data to life, allowing you to visually explore patterns and relationships in your data and, importantly, pull up the underlying data samples with a single click to internalize the underlying data-centric reasons behind a model’s failure mode, statistical outlier, or likely data quality issue.

For example, here’s a panel that displays a histogram of a specified field of the current dataset where clicking a bar loads the corresponding samples in the App:

Walkthrough Panels

You can use a combination of panel components like markdown, buttons, arrow navigation, and layout containers to create guided walkthroughs similar to the ones at https://try.fiftyone.ai.

Here’s an example of a panel that leads the user through multiple steps of a guided workflow:

Building Panels

Python panels are defined using this basic structure:

import fiftyone.operators as foo

class ExamplePanel(foo.Panel):
    @property
    def config(self):
        return foo.PanelConfig(
            name="example_panel",
            label="Examples: Panel",
            icon="extension",  # MUI icon name
        )

    def on_load(self, ctx):
 """Implement this method to set panel state/data when the panel
        initially loads.
        """
        pass

    def render(self, ctx):
 """Implement this method to define your panel's layout and events.

        This method is called after every panel event is executed (panel
        load, button callback, context change event, etc).
        """
        pass

def register(p):
    p.register(ExamplePanel)

The on_load() method is called when the panel initially loads and can be used to initialize the panel’s state and associated larger data.

The render() method defines your panel’s layout, components, and any event handlers. It can make use of the fiftyone.operators.types module, which defines a rich builtin type system that you can use to implement the layout and associated events that define the panel.

An ExecutionContext is passed to each of the panel’s methods at runtime. This ctx contains static information about the current state of the App (dataset, view, panel, selection, etc.) as well as dynamic information about the panel’s current state and data.

Python panels also support a rich set of builtin event handlers that you can implement to respond to in-App events such as view changes. Most panel components support callback methods like on_click() and on_change() that you can implement to perform operations and trigger state updates when users interact with the components:

def on_load(self, ctx):
    # Set initial slider state
    ctx.panel.state.slider_value = 5

def open_compute(self, ctx):
    # Launch an interactive prompt for user to execute an operator
    ctx.prompt("@voxel51/brain/compute_visualization")

def open_embeddings(self, ctx):
    # Open embeddings panel
    ctx.trigger("open_panel", params=dict(name="Embeddings"))

def change_value(self, ctx):
    # Grab current slider value from `ctx.params`
    ctx.panel.state.slider_value = ctx.params["value"]

def render(self, ctx):
    panel = types.Object()

    # Define buttons that work with `on_click` callbacks
    panel.btn(
        "button1",
        label="Compute visualization",
        on_click=self.open_compute,
    )
    panel.btn(
        "button2",
        label="Open embeddings panel",
        on_click=self.open_embeddings,
    )

    # Define a slider with an `on_change` callback
    slider = types.SliderView(
        data=ctx.panel.state.slider_value, label="Example Slider"
    )
    schema = {"min": 0, "max": 10, "multipleOf": 1}
    panel.int(
        "slider_value", view=slider, on_change=self.change_value, **schema
    )

Refer to the docs for full details and examples of building Python panels.

Custom Dashboards

Are you looking for a place to visualize domain-specific analyses or insights into your visual data? Ever wish you could build custom dashboards in FiftyOne to display statistics of interest about your current dataset and the models you’ve trained on it? 

Now you can! Just upgrade to FiftyOne 0.25 and install the new Dashboard Panel:

As you can see above, the Dashboard Panel supports a variety of plot types that you can populate for your datasets:

• Categorical histogram: plot the frequency of each category in a field
• Numeric histogram: plot the distribution of a numeric field
• Line: display a line chart of a numeric field
• Scatter: display a scatterplot of two numeric fields
• Pie: display a normalized pie chart of a categorical field

It also provides a convenient interface to help you quickly define and customize your dashboard’s layout:

• Automatically populates dropdowns of applicable dataset fields when defining plot data sources
• Shows plot previews while constructing new plots to verify correctness before adding them to your dashboard
• Provides a toggle icon in the bottom right to bring the dashboard in/out of editable mode
• Drag to resize and reorder existing plots
• Single click to add/edit/delete plots

You can even define plots backed by custom data sources by writing Python code that queries the current dataset (via the panel’s execution context) or even retrieves information from a third-party API!

SAM 2

Segment Anything 2 (SAM 2) was released on July 29th and represents a major leap forward in zero shot segmentation, both in terms of segmentation quality on images and all-new support for generating segmentation tracks in video sequences using simple prompts—like bounding boxes or keypoints—from a single frame.

With SAM 2 in FiftyOne 0.25, you can now generate and visualize segmentation labels on your image and video datasets with just a few simple commands. Automatically extract insights into the content of your visual data and jumpstart your model training efforts without resorting to expensive and time-consuming large-scale human-in-the-loop labeling efforts.

You’ve got to try these models out for yourself!

Image Segmentation

Here’s how to ask SAM 2 to generate instance segmentations for a given set of bounding box prompts on an image dataset:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart", max_samples=10)
model = foz.load_zoo_model("segment-anything-2-hiera-tiny-image-torch")

# Prompt SAM 2 to generate `segmentations` for `ground_truth` boxes
dataset.apply_model(
    model,
    label_field="segmentations",
    prompt_field="ground_truth",
)

session = fo.launch_app(dataset)

You can also ask SAM 2 to automatically generate segmentations for whole images without any prompts:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart", max_samples=10)
model = foz.load_zoo_model("segment-anything-2-hiera-tiny-image-torch")

# Automatic segmentation
dataset.apply_model(model, label_field="auto")

session = fo.launch_app(dataset)

Video Segmentation

Here’s how to ask SAM 2 to generate segmentation tracks for a video dataset based on a single set of bounding box prompts on the first frame of each video:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video", max_samples=2)
model = foz.load_zoo_model("segment-anything-2-hiera-tiny-video-torch")

# Only retain detections on the first frame of each video
(
    dataset
    .match_frames(F("frame_number") > 1)
    .set_field("frames.detections", None)
    .save()
)

# Prompt with boxes
dataset.apply_model(
    model,
    label_field="frames.segmentations",
    prompt_field="frames.detections",
)

session = fo.launch_app(dataset)

Available SAM 2 Models

SAM 2’s segmentation and video tracking capabilities are very powerful. In the above examples we used the “tiny” models, but all of the following models are available in the FiftyOne Model Zoo:

Image models:

1. segment-anything-2-hiera-tiny-image-torch
2. segment-anything-2-hiera-small-image-torch
3. segment-anything-2-hiera-base-plus-image-torch
4. Segment-anything-2-hiera-large-image-torch

Video models:

1. segment-anything-2-hiera-tiny-video-torch
2. segment-anything-2-hiera-small-video-torch
3. segment-anything-2-hiera-base-plus-video-torch
4. Segment-anything-2-hiera-large-video-torch

Elasticsearch Integration

Elasticsearch is one of the most popular databases for managing unstructured data, and FiftyOne 0.25 introduces a native Elasticsearch integration that makes it easy to use your Elastic cluster as an external vector database to power text and image similarity queries on your FiftyOne datasets. You can now explore your visual data via semantic concepts without the need for explicit labeling or structured metadata.

Once you’ve configured your Elastic backend, connecting to your vector database is as simple as providing the optional backend parameter when creating a new similarity index:

import fiftyone.brain as fob

index = fob.compute_similarity(
    dataset,
    ...,
    backend="elasticsearch",
    brain_key="img_sim",
)

Once the index is created, any text or image similarity queries you perform in the App or SDK will automatically be executed against your Elastic cluster:

Want to learn more about vector search in FiftyOne? Check out this page for a full overview of FiftyOne’s extensive vector search integrations and supported workflows.

Community Updates

Shoutout to the following community members who contributed to FiftyOne 0.25:

• Rustem Galiullin contributed #4569 Add track IDs if present for Ultralytics models
• dimidagd contributed #4509 Allow ragged batches to be configured for custom Torch models

The FiftyOne community continues to grow!

• 3,000+ FiftyOne Slack members
• 8,000+ stars on GitHub
• 30,000+ Meetup members
• 100+ contributors

What’s Next?

• If you like what you see on GitHub, give the project a star.
• Get started! We’ve made it easy to get up and running in a few minutes.
• Join the FiftyOne Slack community, we’re always happy to help.