We gave the FiftyOne Agent one new capability to learn: label a dataset by describing the damage in plain English instead of picking from a fixed class list. This is the story of how we explored that capability, taught it to the agent as a skill, watched the agent hit a real bug on a live dataset, and fixed the skill based on what we saw. Every screenshot in this post comes from the actual session, including the mistake.
Key takeaways
Agentic Labeling is a Beta FiftyOne Enterprise feature that labels images from a natural-language prompt instead of a fixed class list.
It is built around two objects: a reusable agent (prompt, task type, and up to five positive and five negative visual examples) and an asynchronous run created from that agent.
Five task types are supported: Classification, Detection, Caption, Region Classification, and Region Captioning.
Runs can be scoped three ways: all samples, the current view, or the current grid selection. Once live, a run can be paused, resumed, cancelled while keeping its labels, or cooperatively deleted.
The FiftyOne Agent learns new workflows through skills, markdown playbooks that teach it when and how to chain tool calls. A skill-only plugin needs no new code, just a well-written SKILL.md.
The first real run hit a bug. Asked for "only a couple of samples," the run scoped to the entire 1,282-image dataset. The fix was an exact serialization recipe plus a mandatory post-creation scope check.
The dataset, Cable Damage, is self-serve if you want to try this yourself. Link at the end.
What is agentic labeling?
Most auto-labeling tools work from a fixed vocabulary: point a zero-shot model at “car, person, truck” and it finds those things. Agentic Labeling works differently. You write a natural-language instruction like “describe what type of cable damage is visible in this image,” optionally attach a handful of positive and negative visual examples, and a vision-language model does the rest. It’s a Beta feature, and once you look past the single text box, there’s a surprising amount of machinery behind it.
Agents are the reusable, saved half of the system. An agent bundles a text prompt, a task type, an optional list of allowed classes, and up to five positive and five negative visual examples, “label like this” and “don’t label this,” picked straight from the sample grid or uploaded. Agents are listed, fetched, patched, and deleted independently of any run, so a well-tuned prompt becomes a durable asset you can point at a completely different view weeks later, not something you re-type each time.
The five Agentic Labeling task types, what each one labels, and the output it writes. Detection is whole-image only, which is why region tasks are typically chained after a Detection run.
The five Agentic Labeling task types, what each one labels, and the output it writes. Detection is whole-image only, which is why region tasks are typically chained after a Detection run.
Task type
Labels
Output
Object source
Classification
Whole image
One class
N/A
Detection
Whole image
A set of bounding boxes
N/A
Caption
Whole image
Free-text description
N/A
Region Classification
One object at a time
One class per object
Detections field or patches view
Region Captioning
One object at a time
Free-text description per object
Detections field or patches view
Five task types cover most labeling needs, split into two families. Classification, Detection, and Caption label the whole image: one class, a set of boxes, or a free-text description respectively. Region Classification and Region Captioning label one object at a time instead, pulling their objects either from an existing Detections field you point at or from a patches view, and showing the model each object either Cropped (just that patch) or In context (the full image with the object highlighted). Detection is whole-image only, which sets up a neat chaining trick straight from the docs on region tasks: run Detection first to get boxes, then run a region task against that same Detections field to caption or classify each box individually. Coverage runs to frames views of video too, though not yet to 3D, grouped, or multimodal datasets.
The five Agentic Labeling task types: Classification, Detection, Caption, Region Classification, and Region Captioning
Nothing gets saved untested. Before an agent is ever persisted, its Test button runs a real, unsaved inference pass against whichever samples are currently selected in the grid. Click any result thumbnail to open a prediction editor, step through the batch with Previous and Next, hand-correct a box or label, or drop a bad one entirely with Remove from results. None of that touches your dataset, it only shapes what you see, so you iterate on the prompt and examples against real feedback before ever saving.
Runs are where scale and asynchrony come in. A run is created by reference to a saved agent and targets one of three scopes: every sample in the dataset, the current filtered view, or exactly the samples currently selected in the grid. It writes to a label field you choose, and it executes on an always-on, GPU-backed service shared by the whole organization, not a one-shot job you wait on. That means real wall-clock time, including a cold start if the backing model has been idle, and it means a run has a genuine lifecycle to manage: queued, running, completed, failed, cancelled, interrupted, or paused, with live counts for how many samples are done, how many are in flight, and how many failed to parse or infer.
Once a run is live, you have real controls over it, and they are not interchangeable. Pausing stops it in place. Cancelling stops it too, but keeps the run record and whatever labels it already wrote. Resuming a paused run continues it from where it left off rather than re-labeling completed samples; resuming one that failed, was cancelled, or was interrupted starts it over from scratch instead, a distinction worth knowing before you click it. Deleting is cooperative: if a worker is still actively processing that run, delete doesn’t yank it out from under the service, it requests a graceful cancellation instead and only removes the record once nothing is actively writing to it.
Run lifecycle controls in Agentic Labeling. Resume behaves differently depending on why the run stopped, which is the distinction worth knowing before you click it.
Run lifecycle controls in Agentic Labeling. Resume behaves differently depending on why the run stopped, which is the distinction worth knowing before you click it.
Control
Effect
Pause
Stops the run in place
Resume (paused run)
Continues from where it left off, skipping completed samples
Resume (failed, cancelled, or interrupted run)
Starts the run over from scratch
Cancel
Stops the run, keeps the run record and any labels already written
Delete
Cooperative. Requests graceful cancellation if a worker is active, removes the record once nothing is writing
Put together, that’s a full small platform sitting behind one text box: a saved-agent library, a live test-and-edit loop, three ways to scope work, and a run lifecycle with real pause, resume, cancel, and delete semantics, all backed by a shared GPU service. It’s also composable two different ways: chain a Detection run into a region task the way the docs suggest, or drop the same agent in as one stage of a larger multi-stage FiftyOne Workflow alongside human review stages, not just run standalone from its own panel.
The dataset: Cable Damage
Every screenshot in this post comes from the same dataset: 1,282 close-up images of stranded steel cable, each shot against a green backdrop, the kind of imagery you’d get from an automated cable-inspection rig. The dataset already ships with ground-truth detections for known defect types, things like spaced strand and welded strand, so we had a real baseline to compare the agent’s own labeling against.
That made it a good test case for Agentic Labeling specifically because “cable damage” isn’t a fixed vocabulary problem. A frayed strand, a burn mark, and exposed wire all look different, and describing severity and location in a sentence is a much more natural ask than forcing it into a class name.
Step 1: exploring the feature with Playwright
Before writing a single line of the skill, we needed to understand what the agent would actually be driving. We used the Playwright MCP server to log into the live FiftyOne Enterprise app and walk through the entire Agentic Labeling panel by hand: opening it from the samples grid, typing a prompt, picking a positive example from the grid, running a live test, saving the agent, configuring a run’s scope, and watching it complete.
Typing a labeling prompt directly into the Agentic Labeling panel during the Playwright exploration
That hands-on pass surfaced details worth knowing before you write a skill against this feature: the Test button only ever previews whatever samples are currently selected in the grid, not a random sample of the dataset; the one-class cap on Detection lives in the panel, not in the API layer underneath it, so anything scripting against the same operators has to enforce that convention itself; and the panel most people find first, an “AI assistant” template inside the Annotate tab’s workflow canvas, is actually a different, not-yet-available integration marked “Coming soon.” The real entry point is a panel you open from the samples grid.
Step 2: what the detection task-type actually produces
Running a Detection-task agent against the cable dataset made the task type concrete. The model doesn’t just say “there’s damage somewhere,” it returns bounding boxes with labels, the same shape as the dataset’s own ground-truth annotations.
A cable sample with agent-generated bounding boxes labeled “spaced strand” and “welded strand”
Seeing the model’s boxes land in roughly the same places as the dataset’s existing ground truth was the first real signal that the feature worked as advertised, not just on a demo dataset, but on one with defect types specific enough that a generic zero-shot model would have struggled with the vocabulary.
Step 3: writing the SKILL.md
With a working mental model of the feature, we wrote SKILL.md: a markdown playbook that tells the agent when to route a request here (open-ended labeling descriptions, not fixed class lists), which operators to call and in what order, and the failure modes to watch for. The skill covers the full lifecycle: checking the backing service is actually available, routing between Agentic Labeling and the older fixed-class Auto-Labeling feature, filtering the grid to relevant samples before testing, requiring at least one test pass before an agent can be saved, confirming scope before a run starts, and monitoring a run to completion.
The generated SKILL.md file, showing its frontmatter and first Key Directive
We shipped it as a real skill-only plugin addition, no new operator code, just instructions layered on top of operators that already existed. It landed in two repositories that both needed to stay in sync: the FiftyOne Agent’s own plugin repo, and the shared internal skills library other FiftyOne deployments pull from.
Step 4: putting the skill to work
With the skill in place, we opened the FiftyOne Agent against the Cable Damage dataset and asked it the question that started the real end-to-end test:
“Can you write a caption for each image describing what’s wrong with it?”
The FiftyOne Agent’s welcome screen, ready for the first promptThe exact prompt as typed into the agent: “Can you write a caption for each image describing what’s wrong with it?”
The agent routed the request to the new skill, opened the Agentic Labeling panel, and laid out its plan before touching anything: train a new agent with the Caption task type, write a detailed prompt asking for damage type, location, and severity, test it on a few samples first, save it once we approved the test, then come back and run it across the full dataset. It even asked permission before creating anything, in line with the skill’s own directive to confirm before any mutating call.
Step 5: a real bug, caught and fixed
This is the part we’re not going to smooth over, because it’s the most useful part of the story.
Once we approved training the agent, we asked for a first run on all 1,282 images. That run failed outright. We said “not, only a couple of samples” instead, and the agent limited the grid to five and queued a new run. The very first attempt to create that run threw a validation error, [Validation retry 1: .agent_id: Required property], because the agent had nested the operator’s arguments one level too deep. It retried and queued the run correctly on the second try.
Then the run itself misbehaved. Polling its status showed it actively processing, already past forty completed captions, but the total sample count read 1,282, the whole dataset, not the five samples we’d asked for.
The chat and run list showing the scope mismatch: a run named for five samples, actually processing all 1,282
The root cause traced back to two things baked into the skill itself. First, the operator that creates a run expects FiftyOne’s native serialized view-stage format for scoping ({"_cls": "fiftyone.core.stages.Select", "kwargs": [...]}), which is a completely different, incompatible shape from the simplified dialect the app’s own view-filtering tool uses. Second, the skill’s original guidance told the agent that if it wasn’t sure how to build that scope, it should “default to the simplest case, whole dataset or current view,” treating those two as interchangeable when they’re opposites. When the agent couldn’t reconcile the two formats, it fell back to exactly the wrong default.
We rewrote that part of the skill with three fixes, applying a principle straight out of Anthropic’s own skill-authoring guidance: fragile, error-prone operations deserve an exact, low-freedom recipe, not a vague suggestion.
The exact serialization recipe, spelled out literally, with an explicit warning never to reuse the other tool’s dialect and never to fall back to “whole dataset” when unsure.
A disambiguation of the parameter-nesting collision that caused the validation error, with a wrong/right example side by side.
A mandatory scope check, immediately after creating any run that isn’t targeting the whole dataset: fetch the run’s status, compare its reported sample count against what the user actually asked for, and cancel immediately if they don’t match, before ever reporting success.
That last one matters most. It doesn’t just fix the one bug we saw. It means the next time something scopes incorrectly, for any reason we haven’t thought of yet, the agent catches it within seconds instead of quietly running to completion on the wrong data.
The broader version of this problem, how you let an agent act on real systems without it breaking something, is the subject of a talk we gave recently:
The results: captions on 1,282 images
Once the scope was right, the run completed cleanly across all 1,282 images, writing a caption into a damage_description field for every sample.
The completed run’s detail view: Cable Damage dataset, damage_description field, Caption task type
Opening an individual sample showed exactly what the model wrote, sitting right next to the same “spaced strand” and “welded strand” detections from earlier:
Type of Damage: Broken individual wire strands (mechanical failure/fracture).
Location: Upper-right section of the stranded cable.
Severity: Moderate to High; multiple structural strands are completely severed, compromising the cable’s tensile strength.
A sample showing both its bounding-box detections and the agent-written caption describing type, location, and severity of damage
That’s a genuinely useful description: specific about what kind of damage, where it is, and how bad it looks, generated from a single natural-language instruction with no per-class training data.
Bonus: querying the captions you just generated
Free-text captions turn out to be useful for more than reading one at a time. Once every sample had a damage_description, we asked the agent to find the ones that looked most concerning, and it filtered the dataset down to 160 of 1,282 samples whose captions mentioned words like “unusual,” “severe,” “extensive,” or multiple damage types at once, then offered to narrow further or pull specific examples.
The agent filtering the full dataset down to 160 samples with severe or unusual damage descriptions
That’s the natural payoff of writing labels as language instead of fixed categories: the labels themselves become searchable in ways a class name never could be.
Try it yourself
If you want to see Agentic Labeling on this same kind of data without needing our internal demo environment, here’s a self-serve reference, optional, and not part of any live demo:
Agentic Labeling is a Beta FiftyOne Enterprise feature that labels images from a natural-language instruction rather than a fixed class list. You write a prompt like "describe what type of cable damage is visible in this image," optionally attach up to five positive and five negative visual examples, and a vision-language model produces the labels.
Auto-Labeling works from a fixed vocabulary, pointing a zero-shot model at a predefined class list. Agentic Labeling works from an open-ended natural-language description, which suits cases where the thing you are labeling does not fit neatly into a class name.
Five. Classification, Detection, and Caption label the whole image. Region Classification and Region Captioning label one object at a time, pulling objects from an existing Detections field or a patches view.
Not in one pass. Detection is whole-image only, so you chain: run a Detection agent first to get boxes, then run a region task against that same Detections field to caption or classify each box individually.
A skill is a markdown playbook that teaches the FiftyOne Agent when to route a request to a given capability, which operators to call and in what order, and which failure modes to watch for. A skill-only plugin requires no new operator code.
The rewritten skill requires a mandatory scope check immediately after creating any run that does not target the whole dataset. It fetches the run's status, compares the reported sample count against what the user asked for, and cancels the run if they do not match, before reporting success.
Yes. Because the labels are free text, they are queryable as language. In this walkthrough the agent filtered 1,282 samples down to 160 whose captions mentioned terms like "unusual," "severe," or "extensive."
