The Agent Gateway in Voxel51: Let Your Local Agent Work With Your Physical AI Data
by Adonai Vera
• 13 min read
Author

Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera
In this article
Key takeawaysConnect the Voxel51 agent gateway in one promptOr add the MCP server yourselfEither way, here is what happens nextOperators come from your own plugin framework, not a fixed listThe catalog is broad, that is why there is a skillCloud or self-hosted, and more than one deployment at a timeFrom here, your agent is just another way into Voxel51First: what is in this dataset, and how bigSecond: a real data quality scan, with a threshold that lies about its own datasetEvery one of those calls lands in the dataset's own audit trailThird: an agentic labeling run, including the moment it got the dataset's name wrongBuilding your own: the shape that generalizes past this one skillWhere this matters outside a demo orgTry itFrequently asked questions
In this article
Key takeawaysConnect the Voxel51 agent gateway in one promptOr add the MCP server yourselfEither way, here is what happens nextOperators come from your own plugin framework, not a fixed listThe catalog is broad, that is why there is a skillCloud or self-hosted, and more than one deployment at a timeFrom here, your agent is just another way into Voxel51First: what is in this dataset, and how bigSecond: a real data quality scan, with a threshold that lies about its own datasetEvery one of those calls lands in the dataset's own audit trailThird: an agentic labeling run, including the moment it got the dataset's name wrongBuilding your own: the shape that generalizes past this one skillWhere this matters outside a demo orgTry itFrequently asked questions
Try Voxel51 for yourself
Voxel51 just closed the gap between your coding agent and your own Voxel51 deployment. A remote agent gateway connects Claude Code, Cursor, or any agent that supports the Model Context Protocol (MCP) straight to your system, on-premises or in the cloud, with the exact operators and permissions your account already has. Connecting takes one prompt, or one command if you would rather do it by hand, plus a single browser click, and whatever your coding agent does next runs under your own account and your own dataset permissions, logged in the dataset's own activity history the same as if you had clicked it yourself. We set it up twice, from two different clients, against two different real datasets, and used it to run an actual agentic labeling job end to end.
Key takeaways
- The Voxel51 agent gateway is a remote MCP server that connects Claude Code, Cursor, VS Code, GitHub Copilot, and other MCP clients to your own Voxel51 deployment, on Voxel51 Cloud or self-hosted.
- Connecting to the Voxel51 agent gateway takes one prompt or one command, plus a one-time browser approval, with no API key to copy.
- Every Voxel51 agent gateway call runs under the approving user's own role and dataset permissions, re-checked on every call, and appears in the dataset's Activity tab under that user's name.
- The operators available through the Voxel51 agent gateway come from your installed plugins, which added up to 281 operators across 8 plugin namespaces on the deployment tested here.
- On self-hosted Voxel51 deployments, remote agent access sits behind a feature flag that is off by default, so your Voxel51 customer success contact needs to enable it first.
Connect the Voxel51 agent gateway in one prompt
The easiest path needs nothing installed ahead of time. Paste one line into your agent's chat, and it reads a skill file, voxel51-agent-setup, straight off the network and drives the rest of the setup itself.
Read and follow https://raw.githubusercontent.com/voxel51/fiftyone-skills/main/skills/voxel51-agent-setup/SKILL.md to connect me to my Voxel51 deployment.The skill does not just print instructions, it drives the setup. It asks which deployment you are on, Voxel51 Cloud or a self-hosted install, confirms the gateway address, and adds it as a remote server on its own.
Or add the MCP server yourself
None of that is required. The skill is doing exactly what you can do by hand: adding voxel51-agent-gateway as a remote MCP server pointed at your deployment's gateway URL. In Claude Code, that is one command, the same one the skill runs for you:
claude mcp add --transport http --scope user voxel51-agent-gateway \
https://<your-deployment>/api/proxy/fiftyone-teams/mcp \
--client-id voxel51-agent-mcpThe --client-id flag is not optional. This deployment's auth server does not support a client registering its own id automatically, Dynamic Client Registration, so every official client, Claude Code, Claude Desktop, Cursor, VS Code, and GitHub Copilot, pins the same fixed id, voxel51-agent-mcp, rather than inventing one.
A client with no add-server command of its own, Claude Desktop or Cursor among them, takes the same connection as a block of JSON instead:
{
"mcpServers": {
"voxel51-agent-gateway": {
"type": "http",
"url": "https://<your-deployment>/api/proxy/fiftyone-teams/mcp"
}
}
}Either way, here is what happens next
Two things stand out the moment you try to use it. First, a new MCP server only attaches on the next session, so your agent always reports it as unauthenticated until you restart it. Second, there is no API key to copy anywhere: the first real call opens a one-time browser approval screen tied to your own login.
A fresh session later, the same status command reads differently, and the agent is ready for the prompt we actually cared about: an agentic labeling run, which we get back to further down.
Once connected, every client gets the same five tools: list_operators, get_operator_schema, execute_operator, list_delegated_operations, get_delegated_operation. That is the whole surface. list_operators tells you what your account is currently allowed to run, live checked on every call rather than cached, execute_operator runs one, and the delegated operation tools let a long job run in the background while you keep working.
Operators come from your own plugin framework, not a fixed list
list_operators is not a menu Voxel51 ships with the gateway. Operators are plugins, the same extension system that already powers custom panels, workflows, and actions inside the Voxel51 App itself. Whatever your team has installed there is exactly what your coding agent sees through the gateway, nothing more, nothing hardcoded, and nothing that needs a second integration to become agent-callable.
On this deployment, list_operators returned 281 operators spread across 8 plugin namespaces, the general-purpose ones such as @voxel51/operators and @voxel51/workflows doing most of the heavy lifting, plus a handful of more specialized namespaces for things such as agentic labeling and label schemas.
Every one of those is still filtered by the same permission check described above: if your role cannot run an operator in the App, your agent cannot run it through the gateway either, and that stays true the moment your team ships a new one. Write an operator once as a plugin, under your own permissions, on-premises or in the cloud, and it is already something your coding agent can call, no separate agent-facing wrapper to maintain.
The plugin framework operators are built on is documented here: https://docs.voxel51.com/plugins/developing_plugins.html, including how to write and install your own.
The catalog is broad, that is why there is a skill
281 operators is a lot of tool names for an agent to guess at blind, and most of them carry no written description of their own. That is the gap voxel51-agent-setup closes on the other side of the connection: it does not just wire up the gateway, it ships five reference files that tell your agent which real operator to reach for before it ever calls list_operators cold.
- datasets-and-samples.md: dataset lifecycle, views, fields and schema, sample and label edits, indexes, media import, workspaces.
- analysis-and-labeling.md: brain methods, embeddings, model evaluation, data quality, auto-labeling, agentic labeling, annotation, zoo models, workflows, label schemas.
- delegated-operations.md: running things in the background, checking on them, delegated run metadata.
- agent-tools.md: the Voxel51 Agent's own operators, conversations, code execution, plugin and skill authoring, memory.
agent-tools.md covers a different kind of operator entirely: the ones behind the in-app Voxel51 Agent's own chat panel, including write_plugin and write_skill, operators that let an agent draft a new plugin or a new skill from inside a conversation. The same file is also precise about a real boundary: the in-app Agent's own internal memory and its own skill invocation mechanism are hardcoded into its own loop, not operators, and nothing reachable through this gateway touches either one.
Cloud or self-hosted, and more than one deployment at a time
The gateway address has one fixed shape, https://<host>/api/proxy/fiftyone-teams/mcp, and only the host changes. On Voxel51 Cloud that host is always app.voxel51.com. On a self-hosted deployment it is whatever your admin's install is reachable at. The manual setup step in the documentation spells out both cases side by side rather than assuming cloud:
| Deployment | Gateway URL |
|---|---|
| Voxel51 Cloud | https:// |
| Self-hosted Voxel51 | https:// |
One note if you are self-hosted or on-premises specifically: remote agent access ships behind a feature flag, off by default. If it is not already turned on for your deployment, contact your Voxel51 customer success contact to get it enabled before any of the steps above will work.
Registering it by hand, the same claude mcp add command shown above, names the server yourself, which is what makes more than one deployment straightforward: run that command again with a different name and a different host, and your agent has both available side by side.
A team with a staging deployment and a production one, or a consultant who supports several customers' self-hosted installs, adds a second entry the same way under a second name, for example voxel51-agent-gateway-prod next to voxel51-agent-gateway-staging, each pointed at its own host. The agent then asks which one by name instead of needing to be reconfigured every time you switch. The client id stays the same fixed string, voxel51-agent-mcp, for every deployment, because this auth server does not support a client registering itself automatically, every official per-client config, Claude Code, Claude Desktop, Cursor, VS Code, and GitHub Copilot, pins that same id rather than letting the client invent one.
From here, your agent is just another way into Voxel51
Follow the chain so far: you typed one message, your agent read a skill or a config entry, and from there it reached straight into the same platform you would otherwise open in a browser, the same datasets, the same panels, the same permissions. Voxel51 Skills and the Voxel51 MCP gateway are two doors into the same four things underneath: search and visualize, curate and evaluate, annotate, and automate.
If you want to follow along on your own data instead of just reading about ours, Voxel51 Cloud is free to try at app.voxel51.com, nothing to self-host to get started. Before you even bring in your own imagery, the Explore tab is worth a look on its own: real, curated datasets across robotics, manufacturing, healthcare, autonomous vehicles, and more, each one a click away from landing in your own workspace.
The three examples below are what that same chain does once you are actually working, run against two of our own real datasets on a live deployment, two from Cursor in one session, the third from Claude Code's terminal in another. Nothing here is staged for this post, and nothing stops you from pointing the same prompts at a dataset from your own Explore tab.
First: what is in this dataset, and how big
With the gateway connected in Cursor, we asked for exactly what a new dataset deserves before anything else: its field schema and its size.
get_field_schema returned the full schema cleanly, about 70 fields once you count nested classification, detection, and segmentation subfields. The sample count was a different story. There is no dedicated count operator in this gateway's catalog at all, something the skill's own reference files say plainly rather than leaving you to discover it the hard way: no plain count, distinct values, or histogram call exists yet, only execute_code as a fallback for that specific question. We got the number a different way, reading dataset_size back out of a data quality panel call: 2,162 samples.
We also hit a real bug worth naming. get_operator_schema, the tool meant to tell you an operator's exact arguments before you call it, failed on every single operator we tried, including trivial ones such as list_datasets, always with the same error, a schema the tool did not actually return. The workaround was calling execute_operator directly with no arguments and reading the shape of what came back. If your agent hits the same error, that is the gateway, not your setup.
Second: a real data quality scan, with a threshold that lies about its own dataset
Next we asked the agent to scan VisA Anomaly Detection for blurry images, near duplicates, and anything else notable, using the real data quality operators rather than summarizing from memory.
| Check | Flagged | What it means here |
|---|---|---|
| Blurriness | 424 / 2,162 (19.6%) | A real issue, worth a manual pass before training |
| Near duplicates | 2,134 / 2,162 (98.7%) | A threshold artifact, not a cleanup signal, see below |
| Exact duplicates | 0 | None found |
| Aspect ratio outliers | 0 | None found |
| Overexposed / blown out | 0 | None found |
| Blank / low texture | 0 | None found |
The blurriness number is real signal: 19.6% of samples fall under the panel's default threshold, in a range that runs from 8.4 up to 2398.5, heavily skewed toward the blurry end. The near-duplicate number is almost the opposite kind of finding. At the panel's default cosine distance threshold, 98.7% of the dataset gets flagged, which sounds alarming until you remember what VisA actually is: an industrial part inspection dataset, the same part photographed repeatedly under nearly identical lighting. The default threshold answers a question this dataset was never going to answer well. Finding genuinely redundant pairs here needs a tighter cutoff than the one the panel starts you on, not a cleanup pass on the headline number.
Both runs here are Agentic Labeling and Data Quality panel operators, see https://docs.voxel51.com/enterprise/agentic_labeling.html, the same panels a human would click through in the App, just reached through the gateway instead.
Every one of those calls lands in the dataset's own audit trail
This is the part that is easy to miss from a terminal transcript alone: open the same dataset's Activity tab in the App, and Adonai Vera updated 2162 samples via Compute aspect ratio (aspect_ratio, 2162 changes).
That attribution is not cosmetic. The gateway authenticates as the approving user and nothing more, so list_operators only ever shows operators that user's role and dataset permissions currently allow, re-checked on every call rather than cached at connection time. Revoke the pairing, or have an admin remove a permission, and the next call reflects that immediately, the same live check a person clicking through the App would hit.
That live, per-user permission model is the same one documented for roles and dataset-level access generally, see https://docs.voxel51.com/enterprise/roles_and_permissions.html, the gateway does not introduce a separate permission system of its own.
Third: an agentic labeling run, including the moment it got the dataset's name wrong
The last test was the one example from the original prompt list we could run for real end to end: kick off agentic labeling on 20 unlabeled samples and report back when it finishes. We asked for it in Claude Code's terminal, against PhenoBench, a real plant phenotyping dataset already on the deployment.
We typed the dataset name the way people actually type dataset names, lowercase, "phenobench". It did not match. Rather than fail outright or guess, the agent queried the real dataset list, found the actual name, PhenoBench, capitalized differently than typed, and retried against that. Small moment, but it is the live permission and live catalog model from the sections above doing its job: the agent checks what is actually there instead of assuming.
From there the run completed cleanly: 20/20 samples labeled, 0 parse failures, 0 inference failures, using an existing "Crop vs weed" labeling agent already configured on the dataset and the gpt-6.1-sol-responses model, in about 4 minutes.
Everything the terminal reported is checkable in the App itself. The run detail panel shows the same numbers plus ones the terminal summary left out: 114,659 tokens across 20 requests, 77,346 of the input tokens served from cache, about 5,733 tokens per labeled sample.
The labeled samples themselves render the same way any other detections field does in the App, crop in one color, weed in another, laid out across the sugar beet rows the dataset was photographed over.
Long-running operators like this one can run in the background rather than block your agent's reply, see https://docs.voxel51.com/api/fiftyone.operators.delegated.html, which is what delegate=True and the two delegated operation tools are for.
Building your own: the shape that generalizes past this one skill
Nothing about this setup is specific to agentic labeling. The pattern is a skill file an agent fetches and follows, backed by a small, fixed tool surface that stays honest about what your account can currently do. Writing a new skill for your own team's workflow means writing a markdown file that names the real operator URIs for that workflow, the arguments worth knowing up front, and the failure modes a user will actually hit, the same shape as voxel51-agent-setup's own reference files for datasets, analysis and labeling, and delegated operations.
The pattern for writing one is documented directly: https://docs.voxel51.com/agents/developing_skills.html
Where this matters outside a demo org
- Regulated or audited data access: teams in healthcare, finance, or defense contracting often cannot accept a shared service account touching patient, financial, or classified imagery, every access has to trace to a named person. A gateway that authenticates as the approving user and writes every action to that dataset's own activity history is the difference between an auditable agent and one compliance simply will not approve.
- Self-hosted and air-gapped deployments: manufacturing, defense, and government teams frequently run Voxel51 on infrastructure with no path to a public SaaS endpoint at all. The gateway's only requirement is a host your agent can reach and an admin willing to turn the feature flag on, the same flow as the cloud case with one different address.
- Multi-environment pipelines: an ML platform team running separate staging and production Voxel51 deployments, or a consultancy supporting several self-hosted customers, registers each as its own named gateway and tells its agent which one to use, rather than re-pointing a single config file every time the target changes.
Try it
The measurements and transcripts behind every number in this post are in measurements.json in this post's folder, next to the raw transcripts from both sessions.
How to connect an agent to your own Voxel51 deployment: https://docs.voxel51.com/agents/using_agents.html
The voxel51-agent-setup skill itself: https://raw.githubusercontent.com/voxel51/fiftyone-skills/main/skills/voxel51-agent-setup/SKILL.md
To see PhenoBench the way our agent left it, open https://app.voxel51.com/dataset/phenobench/samples, the same dataset this post's agentic labeling run wrote al_plants onto.
Frequently asked questions
The Voxel51 agent gateway is a remote MCP server, voxel51-agent-gateway, that lets a coding agent such as Claude Code or Cursor run operators against your own Voxel51 deployment. It exposes five tools: list_operators, get_operator_schema, execute_operator, list_delegated_operations, and get_delegated_operation.
To connect Claude Code to Voxel51, paste the one-line voxel51-agent-setup prompt into Claude Code, or run claude mcp add with your deployment's gateway URL and the fixed client ID voxel51-agent-mcp. Then restart the session and approve the one-time browser screen on the first call.
No, the Voxel51 agent gateway has no API key to copy. The first call opens a one-time browser approval tied to your own login, and the agent then acts with your permissions until you revoke access.
Yes, the Voxel51 agent gateway works with self-hosted and on-premises deployments at https://<your-deployment>/api/proxy/fiftyone-teams/mcp. Remote agent access ships behind a feature flag that is off by default, so ask your Voxel51 customer success contact to turn it on.
An agent connected through the Voxel51 agent gateway can run any operator your role and dataset permissions allow, including schema reads, data quality scans, and agentic labeling runs. On the deployment in this post, that was 281 operators across 8 plugin namespaces.

Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera
Try Voxel51 for yourself
Related posts
Boost your visual and physical AI workflows.


