Your Agent Is Only As Good As Its Tools

I am Adonai Vera, and I have spent the last months building the FiftyOne Agent at Voxel51. Along the way I kept running into the same realization over and over: the model behind an agent matters much less than people assume. What actually decides whether an agent is useful is the surface of tools, skills, and context you hand it. Give it a small, static toolbox and even the best model will guess. Give it a wide, well organized set of tools and a mediocre model starts to look brilliant. This post is about that idea, and about the agent we built around it.

Key Takeaways

An agent's usefulness is decided by its tools, skills, and context, not by the model behind it.
The FiftyOne Agent is built on two separate pillars: skills (human-authored workflows that teach the agent how) and MCP (a connection layer that lets it actually do things).
The FiftyOne MCP Server exposes 80+ operators for dataset management, model inference, and brain computations, and any operator a plugin registers becomes available automatically.
The agent adds semantic doc search (powered by Kapa AI), three tiers of memory (personal, dataset, org), and bring-your-own-model support across 100+ providers.
Every tool call passes a guardrail check first: run, confirm, or refuse, with unknown-risk operations treated as dangerous by default.

The numbers already tell the story

Look at what happened in coding agents over the last year. Claude Code crossed one billion dollars in annualized revenue within six months of launch, then reached two and a half billion within nine months, according to Anthropic’s own announcement. Weekly active developers doubled in a single month at the start of 2026. Anthropic has said the average developer using Claude Code now spends about twenty hours a week inside the tool, and Claude authored code now accounts for a meaningful share of public GitHub commits.

Google did not sit still either. Antigravity CLI, invoked from the terminal as agy, launched as the successor to Gemini CLI and folded in support for multiple model families behind one command, plus a multi agent mode where a single session coordinates its own sub agents on a task.

Two different companies, two different bets on which model wins, and yet both products converge on the exact same shape: a terminal or IDE agent that can read your repository, call real tools, run real commands, and act instead of just answering. The model race gets the headlines. The tool race is what is actually being won or lost underneath it.

Bringing that lens to computer vision data

That is the exact problem I wanted to solve for FiftyOne. Writing a chatbot that can explain FiftyOne’s API is easy. Writing an agent that can actually open your dataset, run a brain method, evaluate a model, and hand you a working answer, that only happens if the agent has real tools connected to your real data, plus the judgment to know which tool to reach for and when.

So the FiftyOne Agent is built on two pillars that are deliberately kept separate: skills and MCP.

Skills teach the agent how

A skill is a written, human authored workflow. It is not a prompt trick, it is closer to an onboarding document for a new teammate: here is the goal, here are the prerequisites, here is the exact sequence of steps, here is what usually goes wrong. The FiftyOne Agent ships with skills for the workflows people actually run every day.

FiftyOne Agent skills panel showing 13 skills grouped by category (Evaluate, Annotate, Curate, Export, Import, Docs), each with an on/off toggle. — Skills panel showing categories like Evaluate, Annotate, Curate, Export, Import and Docs, each with toggles

In the screenshot above you can see them grouped by what they accomplish. Model Evaluation under Evaluate. Dataset Split, Workflow Management, and Auto Labeling under Annotate. Data Distribution, Data Quality, Data Enrichment, and Dataset Discovery under Curate. Data Export, Data Import, and SDK Guidance rounding things out. Every one of these can be switched on or off per conversation, because not every workflow belongs in every context.

Each skill follows the open Agent Skills specification, a SKILL.md file with frontmatter and sections for overview, prerequisites, key directives, workflow, and troubleshooting. The description field in that frontmatter is the single most important line in the whole file, because it is what the agent reads to decide whether this skill even applies to what you just asked. Plugins declare which skills they ship in fiftyone.yml, and the moment a plugin is installed, its skills are available with no extra wiring.

The skills that ship with the FiftyOne Agent, grouped by the job they do. Each can be toggled on or off per conversation.

The skills that ship with the FiftyOne Agent, grouped by the job they do. Each can be toggled on or off per conversation.
Skill	Category	What it does
Model Evaluation	Evaluate	Evaluates a model and explores results in the Model Evaluation panel
Dataset Split	Annotate	Creates train/val (and optional test) splits using random sampling
Workflow Management	Annotate	Creates, clones, starts, and monitors FiftyOne workflows
Auto Labeling	Annotate	Interprets labeling needs and launches the Auto-Labeling workflow
Data Distribution	Curate	Explores, analyzes, and summarizes a dataset's distribution
Data Quality	Curate	Checks data quality, finds bad images, and detects noise
Data Enrichment	Curate	Enriches a dataset by computing image metadata and signals
Dataset Discovery	Curate	Finds, browses, and loads datasets
Data Export	Export	Exports a dataset or filtered subset to formats like COCO and YOLO
Data Import	Import	Imports media and labels into a FiftyOne dataset
SDK Guidance	Docs	Answers how-to questions about the FiftyOne Python SDK

You can read the full specification for developing your own skills at docs.voxel51.com/agents/developing_skills.html, and the skills that ship by default live in the open fiftyone-skills repository, so nothing here is hidden behind a black box.

MCP gives the agent something real to call: 80+ operators

A skill without a way to execute anything is just good advice. That is where MCP, the Model Context Protocol, comes in. The FiftyOne MCP Server exposes more than eighty operators covering dataset management, model inference, and brain computations, so the agent is not limited to describing what you could do, it can go do it. Any operator registered by a FiftyOne plugin becomes available to the agent automatically. No extra configuration, no separate registration step. Install the plugin, and the agent can use it the same day.

The way the docs put it captures the split cleanly: MCP is about connection, it lets an agent talk to real systems and run real operations, while skills are about guidance, they teach the agent how to use those capabilities correctly for a specific task. One exposes the what. The other explains the how and when. You need both, and I have watched agents fail in both directions when only one is present. All MCP and skills detail lives at docs.voxel51.com/agents/using_agents.html, and the server itself is open source at fiftyone-mcp-server.

The toolbox keeps growing

Eighty operators is not one flat list, it covers a lot of different jobs. The agent can load and summarize a dataset, add fields and samples, build and clear views, filter the sidebar, open and configure panels, select and tag samples, kick off brain runs and evaluations, manage plugins, and adjust color schemes, among plenty more. Each one is a small, well scoped piece of FiftyOne’s own functionality, not a reinvention of it.

One of those tools deserves its own mention. When the agent needs to answer a question about the FiftyOne SDK or docs rather than run an operation, it reaches for a tool that performs semantic search over FiftyOne’s own documentation and returns the most relevant passages along with their source links, and that tool is powered by Kapa AI. It is a small detail, but it means the agent is quoting the actual docs at you instead of recalling something it half remembers from training.

The agent also carries memory across conversations, in three tiers. Personal memory belongs to you and follows you across every dataset you touch. Dataset memory is scoped to whatever you are working on right now. Org memory is shared across your whole team. None of it is invisible, it is just another kind of tool the agent reaches for when it decides something is worth remembering past the current chat.

Bring your own model, seriously

Because the model is the least differentiating part of this system, we did not want to lock anyone into one provider. The agent supports over one hundred providers including Anthropic, OpenAI, and Google, plus custom endpoints for teams running their own enterprise gateway.

FiftyOne Agent add-provider settings screen with fields for name, provider, endpoint, API key, allowed models, custom model names, and extra headers. — Add provider settings screen showing name, provider, endpoint, API key, allowed models, custom model names, and extra headers fields

You can point the endpoint at an internal proxy, prefix model names with the provider slug, for example openai/your-model-id, and attach extra headers for authentication. Every request also carries an X-FiftyOne-User-Email header automatically, so usage stays attributable per person even behind a shared gateway. Full setup details are in the enterprise docs at docs.voxel51.com/enterprise/agent.html.

Instructions are just as flexible. You can set user level instructions that apply to every conversation you personally have with the agent.

FiftyOne Agent settings showing the User instructions field, which inserts custom guidance into every conversation with the agent. — Agent Settings screen showing the User instructions text field, described as inserted into every conversation you have with the agent

There are also system level and dataset level instructions, so an org can set baseline behavior while individuals and even individual datasets layer their own preferences on top.

Watching it actually write and run code

None of this matters if the agent cannot produce working code against your real data, so here is exactly what that looks like in practice.

FiftyOne App with the agent panel open, showing generated Python SDK code that clones the predictions field to prediction_duplicate, with a Run button before execution. — FiftyOne App with the agent panel open, having written Python SDK code to clone the predictions field to prediction_duplicate, with the run button visible and the result confirming the new field was created

I asked the agent to clone a field on a loaded dataset. It wrote this, using FiftyOne’s own SDK, not a guess at some generic pandas syntax:

The code shows up in the chat as a card with Run and Edit buttons, so nothing executes until you say so. I hit Run, and a moment later the dataset had a new field called prediction_duplicate, confirmed in the same reply. That pause before execution is not an afterthought, it is the whole point of pairing tools with judgment: the agent proposes, you approve, and only then does anything touch your data.

Here is the same idea captured live, filtering a real 210 sample dataset straight from the chat panel.

A frame from the demo video, showing the agent panel mid conversation, having just filtered a dataset down from a request typed in plain language

The full recording sits alongside this document as agent_action.mp4, worth watching end to end if you want to see the whole exchange, from the plain language request to the filtered result rendering in the grid.

It remembers where you left off

Every conversation is saved, searchable, and easy to pick back up, and it is private to you. Conversations are scoped to your own identity, so a teammate working on that very same dataset does not see your chat history by default.

FiftyOne Agent chat history panel listing saved past conversations, including writing Python SDK code, printing sample counts, and filtering samples. — Chat history panel listing past conversations, including one about writing Python SDK code to duplicate predictions, one about printing sample counts, one about filtering samples, and one about ground truth labels

That persistence turns the agent from a one shot question box into something closer to a running collaborator on your dataset, one you can return to a day later without re explaining what you were doing.

Every tool call passes through a guardrail first

Handing an agent eighty plus operators only works if it cannot quietly do something destructive with them. Before any tool call actually runs, it passes through a guardrail check that decides one of three things: run it immediately, ask you to confirm first, or refuse outright. A handful of operations are blocked unconditionally, no matter who is asking. Anything that would change what you are looking at in the App, like a view or a filter, always asks first. Anything with no declared risk level is treated as dangerous by default rather than assumed safe. And the agent can never reach into a dataset outside the one you already have open in that session.

That is the short version. Guardrails and the sandboxing around them are big enough topics that they deserve their own post, so consider this a preview. A longer write up on how we think about agent safety, confirmation flows, and sandboxing is coming next.

The actual takeaway

Go back to the opening numbers. Claude Code and Antigravity CLI are not winning because one model is smarter than the other in some abstract sense. They are winning because they gave developers a huge, growing surface of tools to reach for, wrapped in enough guidance that the agent usually reaches for the right one. That is the entire design philosophy behind the FiftyOne Agent too. Skills teach it how to work with computer vision data the way an experienced FiftyOne user would. MCP and its eighty plus operators, plus a Kapa powered search over our own docs, give it real capabilities to call on top of your real datasets. Memory lets it carry context past a single conversation. Guardrails keep all of that honest, so none of it runs without the right level of confirmation. Bring your own model support means you are never boxed into a single vendor’s roadmap.

An agent is not judged by which model sits behind it. It is judged by how much of your actual workflow it can reach, and how well it has been taught to use what it can reach. Everything above, the skills system, the MCP server, the enterprise gateway support, is documented in the open at docs.voxel51.com, and the underlying skills and MCP server repositories are public on GitHub if you want to see exactly how a tool call turns into a result on your screen. If you are curious where this goes next, that documentation is the best place to watch it happen.

This is still early, and we want to hear from you

Everything described above is real and shipping today, but none of it is finished. New skills get added as we watch how people actually use the agent. New operators show up as FiftyOne itself grows. The guardrail policies get tuned as we learn where confirmation genuinely helps and where it just gets in the way. None of that improves without hearing from the people actually running these workflows day to day.

If a skill missed the mark, if the agent reached for the wrong tool, if you hit a wall it should have known how to get around, tell us. That kind of feedback shapes what gets built next far more than any roadmap we could write on our own. We are building this in the open, and we would much rather hear what is missing from you than guess at it ourselves.

FAQ

What is the FiftyOne Agent?

An in-app agent for FiftyOne that can open your dataset, run brain methods, evaluate models, and write and run FiftyOne SDK code against your real data, rather than only describing what you could do.

What is the difference between skills and MCP in the FiftyOne Agent?

Skills are human-authored workflows that teach the agent how and when to do a task. MCP is the connection layer that lets it actually run real operations. Skills provide guidance, MCP provides capability, and the agent needs both.

How many operators does the FiftyOne MCP Server expose?

More than eighty, covering dataset management, model inference, and brain computations. Any operator registered by a FiftyOne plugin becomes available automatically, with no extra configuration.

Which model providers does the FiftyOne Agent support?

Over one hundred providers, including Anthropic, OpenAI, and Google, plus custom endpoints for teams running their own enterprise gateway.

How does the FiftyOne Agent keep tool calls safe?

Before any tool call runs, a guardrail check decides to run it, ask for confirmation, or refuse. Some operations are blocked unconditionally, view and filter changes always ask first, and the agent can never reach a dataset outside the one open in your session.

Does the FiftyOne Agent remember past conversations?

Yes. Conversations are saved, searchable, and private to you, and the agent carries three tiers of memory: personal, dataset, and org.

Your Agent Is Only As Good As The Tools You Give It

Talk to an AI expert