Google shipped Gemini Omni for video generation and agentic video understanding for reading footage back. We built both into FiftyOne as one plugin, then shipped skills beside it so the FiftyOne Agent knows when and how to use them. The same pattern works for whatever your team builds.
Every team has something only they need. A model trained on your own data. An internal service that scores samples the way your reviewers do. A validation step your regulator asks for. A vendor API your competitors do not use. None of it ships with any platform, because none of it generalizes past you.
Key takeaways
FiftyOne Agent skills are Markdown files shipped inside a FiftyOne plugin that tell the agent when and how to use the plugin's operators.
Enabling a plugin makes its operators visible to the FiftyOne Agent immediately, with no retraining or redeploy.
The gemini-vision plugin adds Gemini Omni video generation and agentic video understanding to FiftyOne as one plugin with four skills.
Gemini video understanding results are written as TemporalDetections, so timestamps are seekable and filterable rather than trapped in a paragraph.
On a 30-minute repetitive haystack, static sampling at 0.5 fps matched agentic mode's accuracy at roughly one ninth the token cost.
You can already build that as a FiftyOne plugin, and teams do. What is new is what happens next. The FiftyOne Agent sees your operator the moment the plugin is enabled in your deployment, and can run it. No retraining, no redeploy, nothing to change on the agent side. Your specific, local, unglamorous integration becomes something the agent can reach for.
Being able to run something is not the same as knowing when to. The agent can see your operator exists without knowing which of its options fits the question in front of it, what a run costs, which parameters are safe to guess, or what to do with the result. That gap is what skills close. A skill is a Markdown file you ship inside your own plugin, written by the people who know how the tool should be used, and the agent reads it before acting.
So you are not just handing the agent a new capability. You are handing it your team's judgment about that capability. To show what that looks like end to end, we built something FiftyOne genuinely could not do before: Google's two new video models, generation and understanding, as one plugin with four skills.
Clips generated by Gemini Omni, sitting in a FiftyOne dataset as ordinary video samples, each carrying the prompt that made it.
A plugin is the tool. A skill is the know how.
What a FiftyOne plugin and a FiftyOne Agent skill each give the agent, and how the agent discovers each one.
What a FiftyOne plugin and a FiftyOne Agent skill each give the agent, and how the agent discovers each one.
What it gives the agent
How the agent gets it
Plugin
The ability to do something new
Operators appear in list_operators() as soon as the plugin is enabled
Skill
Knowing when and how to do it well
SKILL.md files declared in fiftyone.yml, discovered from the manifest
Without the skill, an agent handed a video operator with ten tasks has to guess. With it, the agent reads which phrasing maps to which task, that it should draft at low resolution before spending real money, that generated media must be tagged synthetic so it never slips into a training split, and what to do when a run fails. You are shipping instructions along with the capability.
The rest of this post is one worked example: adding Google's two new video capabilities to FiftyOne, then teaching an agent to drive them.
What we added: Gemini video generation
Gemini Omni became generally available on August 27, 2026. It takes text, images, or video and returns an MP4 with a synthesized audio track. A single generation runs 3 to 10 seconds and can be extended in 10 second steps up to 40 seconds. Drafts render at 360p for roughly a third of the cost of 720p, and finals upscale to 4K.
The plugin exposes six tasks, picked by what you are starting from:
The six Gemini Omni generation tasks in the plugin, what each starts from, and what you need selected in FiftyOne to run it.
The six Gemini Omni generation tasks in the plugin, what each starts from, and what you need selected in FiftyOne to run it.
Task
Starting from
Selection
text_to_video
a description
none
image_to_video
one image to animate
1 image
first_last_frame
a start and an end frame
2 images
reference_to_video
subjects to keep consistent
1 to 3 images
edit
a clip the plugin generated
1 generated video
extend
a clip the plugin generated
1 generated video
The six generation tasks, each labelled with what it needs selected.
The generated clip is not a file you download and lose track of. It is added to your dataset as a video sample carrying the prompt that produced it, the task used, the model, the resolution, and the IDs of the samples it drew on.
The interaction ID is the feature
One field on that sample matters more than the rest: gemini_interaction_id.
Omni's editing is stateful. You do not send it a video and a diff. You resume the interaction that produced the video and describe what should change, and the model keeps everything you did not mention. That is the difference between iterating on a shot and rolling the dice again with a longer prompt.
So the plugin stores that id on the sample. A clip generated on Monday is still editable on Wednesday, by anyone with access to the dataset, because the handle to the model's own state travels with the data:
Those two calls turned a daylight warehouse into a night shot, then turned a 10 second clip into 20. Each result is itself editable, so a session converges on a shot over several turns instead of restarting.
The honest limit: this only works on clips the plugin generated. Footage you imported has no interaction to resume, and the operator says so plainly instead of failing in a confusing way.
The generation operator. Draft at 360p, then re render once the shot is right.
What we added: reading video back as data
The other direction is agentic video understanding, where the model navigates the timeline itself rather than being handed a fixed strip of frames.
Here the design decision that mattered has nothing to do with the model. Ask any vision language model what happens in a video and you get a paragraph. You cannot filter a paragraph, sort by it, or seek to it. So instead of asking Gemini to describe what it found, each task asks for a JSON schema of timestamped events: start, end, label, description, confidence.
Structured timestamps become TemporalDetections, written straight onto the video sample, where they render as segments on the App timeline and filter like any other label. Eight of the ten tasks work this way.
Gemini's answers as temporal detections, seekable in the timeline and filterable in the sidebar.
That last line matters more than the number. Almost every disagreement about a count is really a disagreement about what counts as one, so we ask the model to state its rule and put it in front of the user.
How the FiftyOne Agent finds plugin skills
Everything above is an operator, and an operator is only useful to someone who knows it exists. That is the same problem an agent has.
FiftyOne plugins can ship skills: Markdown workflow guides declared in the plugin's own manifest. The FiftyOne Agent discovers them from the manifest as soon as the plugin is enabled. There is no registration step and nothing to change on the agent's side.
The body is not documentation. It is operating instructions: which task to pick from which phrasing, how to bound the cost, what to do with the result, and what the failure modes look like.
One mechanism is worth calling out. The agent parses each skill's operator table and reads a Requires Prompt column. Operators marked False it may run directly. Operators marked True always surface their parameters first, so the user can review and edit before anything runs or bills. Every operator in this plugin is marked True, because each call costs money.
The agent using it
With the plugin enabled, this is a complete instruction:
How many distinct vehicles are in this clip?
Nothing in that sentence names an operator, a task, or a parameter. The agent finds the skill from the plugin's manifest, reads that counting is what the count task does, and assembles the call itself.
The agent resolves a plain question to the right operator and task, on the user's current selection.
Read what it proposes closely. It picked the count task, wrote a prompt spelling out its own counting rule, named a label field, and set a thinking level and a one video cap. The user specified none of that.
The proposed call, surfaced for review before anything runs or bills.
This is the part that only works because of the skill. A bare operator gives the agent a button. The skill is what tells it which of the ten tasks fits the question, to bound the run to one video, and to write a descriptive label field so results stay findable later.
And the result lands as data, not as a chat message:
armored_truck [1, 120], silver_car [1, 31], white_suv [60, 120]. The answer as temporal detections on the sample, seekable and filterable.
Point the same agent at generation and the other skill takes over. Here it generated a warehouse clip with Omni and tagged it synthetic without being asked, because the skill instructs that so a generated sample never drifts into a training split unmarked. It also reported honestly when one of its two background jobs did not produce the sample it expected.
Generation running as background jobs, the clip tagged synthetic, and the agent flagging its own failed run rather than claiming success.
What we measured
On the understanding side we did not want to repeat Google's efficiency figures, so we built a test where the right answer is checkable. Clips from FiftyOne's quickstart-video dataset were concatenated into one long video, with the single clip containing an armored truck inserted exactly once at a known offset. Then the same question, of the same video, under four processing configurations.
On the 30 minute version, at 163 MB, uploaded through the Files API:
Four processing configurations on the same 30-minute needle-in-a-haystack video (163 MB, Files API). Static 0.5 fps found the target at roughly one ninth the tokens of agentic high.
Four processing configurations on the same 30-minute needle-in-a-haystack video (163 MB, Files API). Static 0.5 fps found the target at roughly one ninth the tokens of agentic high.
Configuration
Tokens
Video lookups
Seconds
Found the truck
Agentic, thinking_level="high"
553,560
18
145.8
yes, exactly
Agentic, thinking_level="low"
274,183
16
72.3
no, missed it
Static, fps=1
119,184
0
17.8
yes, exactly
Static, fps=0.5
61,053
0
15.5
yes, exactly
The cheapest run was also exactly right. The second most expensive was the only one that missed.
Two findings, neither of them what we expected. Agentic mode has a cost floor, because the loop spends tokens deciding where to look before it looks. On footage that a uniform pass already handles, static sampling at half a frame per second was roughly nine times cheaper and returned the same timestamps to the second. And thinking_level low cost us recall, not just latency. On the long video it was the only configuration that missed a target the other three found exactly.
The caveat is real. Our haystack is visually repetitive by construction, which is close to the best case for uniform sampling. Google's figures are for long, varied content where most of the timeline is genuinely irrelevant. We have not refuted them. We have established that they are not automatic. So the plugin ships both modes, puts this benchmark in the skill the agent reads, and tells you to try static first.
Building your own FiftyOne Agent skill
If you take one thing from this, take the shape rather than the integration. Any capability you can reach from Python can become an operator, and any operator can become something your agent uses well:
Write the operator. Whatever your team already does by hand. It shows up in the agent's tool list the moment the plugin is enabled.
Return data, not prose. Ask your model for a schema and write real FiftyOne labels. An answer you can filter is worth more than a paragraph you have to re read.
Ship a SKILL.md beside it. Say when to use it, how to keep it cheap, what to do with the result, and how it fails. This is the part that turns a button into judgment.
Decide what the agent may do alone. The Requires Prompt column is where you draw that line, and it is worth thinking about before your agent spends money for you.
Find every time a pedestrian appears in these videos
Why does the car slow down?
Pull the best frames out of this video
Generate a clip of a forklift in a warehouse, then make it night time
The agent reads the skills, picks the task, shows you the call it intends to make, and waits. You approve or edit it. The answer comes back as labels on your samples, not as text in a chat window that you have to copy somewhere useful.
That last prompt is the one to try if you only try one. It is two tasks in one sentence, and a naive agent generates twice. This one generates once, then edits what it made, because the skill tells it that editing preserves the shot you already liked.
The operators, the four skills, and the schemas behind them are all here: https://github.com/AdonaiVera/gemini-vision-plugin. Fork it, swap Gemini for whatever your team already uses, and your agent picks up the new tool the moment you enable the plugin.
FAQ
A skill is a SKILL.md file declared in a plugin's fiftyone.yml that gives the FiftyOne Agent operating instructions for the plugin's operators: which task fits which phrasing, how to bound cost, what to do with the result, and how it fails.
It reads them from the plugin manifest as soon as the plugin is enabled, with no registration step and nothing to change on the agent side.
Only if the skill's operator table marks it Requires Prompt = False. Every operator in the gemini-vision plugin is marked True, so the agent surfaces its proposed call for review before anything runs or bills.
No. Omni editing resumes a stored interaction, so it only works on clips the plugin generated. The operator says so instead of failing.
Not automatically. On a 30-minute repetitive test video, static sampling at 0.5 fps found the target exactly at about one ninth the tokens of agentic high thinking, and agentic low thinking was the only mode that missed.
