Your Robot Data Needs Custom Tools. Build Them as FiftyOne Plugins

Key takeaways

• A FiftyOne plugin is a folder with two files, fiftyone.yml and __init__.py. FiftyOne discovers it at runtime, so you never fork FiftyOne or wait for a release to add a button, form, panel, or background job.
• FiftyOne reads .mcap, .bag, and .rrd files as media_type="multimodal" episodes and imports LeRobot v3 with fo.types.LeRobotDataset, so a plugin can score every episode in a dataset with the same code.
• FiftyOne plugin code runs on the FiftyOne server in Python so that it can use NumPy, SciPy, PyTorch, a GPU, or a secret API key. Long jobs run in the background through delegated execution.
• Rerun extends its viewer in Rust only, with interfaces its docs say break every release. Foxglove extensions are TypeScript running in the browser against one recording. Encord custom agents are HTTPS endpoints tied to labeling workflow stages. None of the three gives you server-side Python tools with custom UI that operate on a whole dataset.
• Two open-source plugins show the pattern for physical AI: lerobot_data_curation ranks LeRobot episodes worst-first, and demo_quality_scorer does the same for MCAP recordings.

Why physical AI data needs custom tooling

• Is the motion in this episode smooth, or full of corrections?
• Did any sensor drop frames, drift its clock, or clip its values?
• Does the action track the state, or is there lag between video and action?
• Is the gripper behavior consistent with the other episodes of the same task?
• Is the language instruction usable for fine-tuning a vision-language-action (VLA) model?

What is a FiftyOne plugin?

• Operators are actions. You trigger one, it runs, it finishes. "Compute episode quality" is an operator.
• Panels are UI that stays open next to your data and reacts to what you do. A worst-first ranking that re-sorts when you filter the grid is a panel.

The anatomy of a FiftyOne plugin

• fiftyone.yml is the manifest. It holds the plugin name in @org/name form, the operators and panels it exposes, and any secrets it needs.
• __init__.py holds your classes and a register(p) function.
• requirements.txt lists pip dependencies. For MCAP work, that's usually mcap, mcap-protobuf-support and friends.
• Everything below the dashed line is optional JavaScript for custom UI.

A complete operator in 25 lines

import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.operators.types as types
from fiftyone import ViewField as F


class TagLowConf(foo.Operator):
    @property
    def config(self):
        return foo.OperatorConfig(name="tag_low_conf", label="Tag low-confidence")

    def resolve_placement(self, ctx):
        return types.Placement(
            types.Places.SAMPLES_GRID_SECONDARY_ACTIONS,
            types.Button(label="Tag low-conf"),
        )

    def resolve_input(self, ctx):
        inputs = types.Object()
        fields = list(ctx.dataset.get_field_schema(embedded_doc_type=fo.Detections))
        inputs.enum("field", fields, required=True, label="Label field")
        inputs.float("thresh", default=0.3, label="Confidence threshold")
        return types.Property(inputs)

    def execute(self, ctx):
        field, thresh = ctx.params["field"], ctx.params["thresh"]
        view = ctx.target_view().filter_labels(field, F("confidence") < thresh)
        view.tag_samples("low_conf")
        ctx.ops.notify(f"Tagged {len(view)} samples")


def register(p):
    p.register(TagLowConf)

1. config names the operator. name is its ID, and label becomes the form's heading.
2. resolve_placement puts a "Tag low-conf" button in the grid's action bar.
3. resolve_input declares a dropdown and a number. The App builds the form, with defaults and validation, from those two lines.
4. execute does the work on the target view and sends a notification.

How an operator call works

1. Resolve inputs. You click the button. The App sends the current context (dataset, view, selection) to the server, which calls your resolve_input and returns a schema. The App turns it into a form. If the operator is dynamic, this loop re-runs on every input change, which is how demo_quality_scorer shows a signal picker only after you choose a telemetry channel.
2. Execute. You hit Execute. The server validates the parameters against your schema, then calls execute. You return a result plus any UI operations, and the App applies them.

Long jobs run in the background with delegated execution

Panels: UI that stays open

class ClassCounts(foo.Panel):
    @property
    def config(self):
        return foo.PanelConfig(name="class_counts", label="Class counts")

    def on_load(self, ctx):
        self.refresh(ctx)

    def on_change_view(self, ctx):
        self.refresh(ctx)

    def refresh(self, ctx):
        counts = ctx.view.count_values("ground_truth.detections.label")
        ctx.panel.set_state("n", len(counts))
        ctx.panel.set_data("counts", [{"type": "bar", "x": list(counts), "y": list(counts.values())}])

    def render(self, ctx):
        panel = types.Object()
        panel.md(f"**{ctx.panel.get_state('n')} classes**")
        panel.plot("counts")
        return types.Property(panel)

Operators compose

How FiftyOne plugins compare to other extension points

Two FiftyOne plugins for MCAP and LeRobot curation

LeRobot Data Curation

• A typed form with tabs. The compute operator's form has Data, Metrics, Camera, and Normalization tabs. You pick the observation.state and action arrays, and assign arm and gripper joints by name. Nothing is guessed from the data, and your picks are remembered for the next run.
• Delegated by default. The operator sets allow_delegated_execution=True and default_choice_to_delegated=True, so scoring runs in the background. Camera decoding is the slow part.
• A JS panel backed by Python operators. The LeRobot Curation panel is a React component with Overview, Motion & Action, Integrity & Coverage, Vision and Language tabs. Click a histogram bar to filter the grid to those episodes. Click a row to open an inspector with joint traces, the speed profile, the gripper timeline, and frames picked around the flagged spans.
• It follows the view. The panel refreshes when the view changes, so filtering the grid re-ranks it.
• Tags on the timeline. Flagged spans such as idle stretches, the longest pause, and acceleration spikes are written as temporal tags on each episode so that you can scrub straight to them in the multimodal viewer. Footer buttons apply review, exclude-candidate, and relabel sample tags.

import fiftyone as fo

dataset = fo.Dataset.from_dir(
    dataset_dir="/data/lerobot/my_dataset",
    dataset_type=fo.types.LeRobotDataset,
)
session = fo.launch_app(dataset)
# then run "LeRobot curation: compute quality" from the operator browser

Demo Quality Scorer

1. Motion smoothness: SPARC, log dimensionless jerk (LDLJ), jerk RMS, and a low-to-high frequency power ratio, computed over windows of a speed profile you choose from position or velocity signals such as /odom -> twist_linear.
2. Sensor health: dropout, rate stability, clock drift, clipping, and cross-channel desync, all from message timestamps and raw values, so it works on any channel the plugin can decode.
3. Outliers: models that flag episodes whose motion and health features don't look like the rest of the batch.

• A dynamic form. The form inspects the first sample's channels. If no telemetry channel carries a numeric signal, the motion family switches itself off and says why. If the server lacks a decoder, the form names the package to install.
• Immediate or delegated, your choice. Run immediately with an in-app progress bar, or schedule it in the background (default).
• A JS panel. The Episode Quality panel is a React component that has Motion, Health, and Outliers tabs. Click a histogram bar to filter the grid. Click a table row to open that episode in the multimodal viewer, with a toast naming the timecode of its worst flagged interval.
• Results as fields. Scores land in quality.* fields such as quality.sparc and quality.ldlj, so you can sort and filter on them outside the panel too.

pip install "fiftyone[multimodal]>=1.19.0" mcap mcap-protobuf-support \
    mcap-ros1-support mcap-ros2-support numpy scipy scikit-learn
fiftyone plugins download https://github.com/harpreetsahota204/demo_quality_scorer

Build your own in six steps

Step 1: Scaffold the plugin

name: "@your-org/gripper_checks"
description: Flags episodes with repeated grasp attempts
operators:
  - score_grasps

Step 2: Write the operator

Step 3: Register it

Step 4: Run the App in debug mode

export VFF_MULTIMODAL=1
fiftyone app debug my_lerobot_dataset

Step 5: Iterate

Step 6: Ship it

fiftyone plugins download https://github.com/your-org/gripper_checks
fiftyone plugins requirements @your-org/gripper_checks --install

Try it yourself

• Plugin developer guide
• FiftyOne multimodal user guide for MCAP and LeRobot
• voxel51/fiftyone-plugins, the official plugins and panel examples
• lerobot_data_curation
• demo_quality_scorer
• FiftyOne MCP server
• VOODO, the React design system for plugin panels