The Agent Gateway in Voxel51: Let Your Local Agent Work With Your Physical AI Data

•
13 min read
Author
Headshot of Adonai Vera
Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera

Try Voxel51 for yourself

Voxel51 just closed the gap between your coding agent and your own Voxel51 deployment. A remote agent gateway connects Claude Code, Cursor, or any agent that supports the Model Context Protocol (MCP) straight to your system, on-premises or in the cloud, with the exact operators and permissions your account already has. Connecting takes one prompt, or one command if you would rather do it by hand, plus a single browser click, and whatever your coding agent does next runs under your own account and your own dataset permissions, logged in the dataset's own activity history the same as if you had clicked it yourself. We set it up twice, from two different clients, against two different real datasets, and used it to run an actual agentic labeling job end to end.

Key takeaways

  • The Voxel51 agent gateway is a remote MCP server that connects Claude Code, Cursor, VS Code, GitHub Copilot, and other MCP clients to your own Voxel51 deployment, on Voxel51 Cloud or self-hosted.
  • Connecting to the Voxel51 agent gateway takes one prompt or one command, plus a one-time browser approval, with no API key to copy.
  • Every Voxel51 agent gateway call runs under the approving user's own role and dataset permissions, re-checked on every call, and appears in the dataset's Activity tab under that user's name.
  • The operators available through the Voxel51 agent gateway come from your installed plugins, which added up to 281 operators across 8 plugin namespaces on the deployment tested here.
  • On self-hosted Voxel51 deployments, remote agent access sits behind a feature flag that is off by default, so your Voxel51 customer success contact needs to enable it first.

Connect the Voxel51 agent gateway in one prompt

The easiest path needs nothing installed ahead of time. Paste one line into your agent's chat, and it reads a skill file, voxel51-agent-setup, straight off the network and drives the rest of the setup itself.
Read and follow https://raw.githubusercontent.com/voxel51/fiftyone-skills/main/skills/voxel51-agent-setup/SKILL.md to connect me to my Voxel51 deployment.
That single line, pasted into Claude Code. No install step runs first, the agent fetches the skill itself and takes it from there.
The skill does not just print instructions, it drives the setup. It asks which deployment you are on, Voxel51 Cloud or a self-hosted install, confirms the gateway address, and adds it as a remote server on its own.
The skill asking which deployment this is, confirming the gateway URL, adding the server, and explaining the restart and one-time approval that come next.

Or add the MCP server yourself

None of that is required. The skill is doing exactly what you can do by hand: adding voxel51-agent-gateway as a remote MCP server pointed at your deployment's gateway URL. In Claude Code, that is one command, the same one the skill runs for you:
claude mcp add --transport http --scope user voxel51-agent-gateway \
  https://<your-deployment>/api/proxy/fiftyone-teams/mcp \
  --client-id voxel51-agent-mcp
The --client-id flag is not optional. This deployment's auth server does not support a client registering its own id automatically, Dynamic Client Registration, so every official client, Claude Code, Claude Desktop, Cursor, VS Code, and GitHub Copilot, pins the same fixed id, voxel51-agent-mcp, rather than inventing one.
A client with no add-server command of its own, Claude Desktop or Cursor among them, takes the same connection as a block of JSON instead:
{
  "mcpServers": {
    "voxel51-agent-gateway": {
      "type": "http",
      "url": "https://<your-deployment>/api/proxy/fiftyone-teams/mcp"
    }
  }
}

Either way, here is what happens next

Two things stand out the moment you try to use it. First, a new MCP server only attaches on the next session, so your agent always reports it as unauthenticated until you restart it. Second, there is no API key to copy anywhere: the first real call opens a one-time browser approval screen tied to your own login.
Right after adding the gateway, Claude Code's own /mcp status still reads needs authentication, not authenticated. This is expected, not a failure.
The one-time browser screen. It names the exact client, voxel51-agent-mcp, asking to act on your behalf with your own permissions until you revoke access.
Approve once, close the tab, and the agent picks the connection up on its own.
A fresh session later, the same status command reads differently, and the agent is ready for the prompt we actually cared about: an agentic labeling run, which we get back to further down.
Authentication successful, connected to voxel51-agent-gateway, in the same terminal that reported needs authentication a moment earlier.
Once connected, every client gets the same five tools: list_operators, get_operator_schema, execute_operator, list_delegated_operations, get_delegated_operation. That is the whole surface. list_operators tells you what your account is currently allowed to run, live checked on every call rather than cached, execute_operator runs one, and the delegated operation tools let a long job run in the background while you keep working.

Operators come from your own plugin framework, not a fixed list

list_operators is not a menu Voxel51 ships with the gateway. Operators are plugins, the same extension system that already powers custom panels, workflows, and actions inside the Voxel51 App itself. Whatever your team has installed there is exactly what your coding agent sees through the gateway, nothing more, nothing hardcoded, and nothing that needs a second integration to become agent-callable.
Voxel51's plugin architecture. An operator is one piece of this system, the same system panels and workflows are built on, which is why a plugin your team already installed shows up through the gateway automatically.
On this deployment, list_operators returned 281 operators spread across 8 plugin namespaces, the general-purpose ones such as @voxel51/operators and @voxel51/workflows doing most of the heavy lifting, plus a handful of more specialized namespaces for things such as agentic labeling and label schemas.
Every one of those is still filtered by the same permission check described above: if your role cannot run an operator in the App, your agent cannot run it through the gateway either, and that stays true the moment your team ships a new one. Write an operator once as a plugin, under your own permissions, on-premises or in the cloud, and it is already something your coding agent can call, no separate agent-facing wrapper to maintain.
The plugin framework operators are built on is documented here: https://docs.voxel51.com/plugins/developing_plugins.html, including how to write and install your own.

The catalog is broad, that is why there is a skill

281 operators is a lot of tool names for an agent to guess at blind, and most of them carry no written description of their own. That is the gap voxel51-agent-setup closes on the other side of the connection: it does not just wire up the gateway, it ships five reference files that tell your agent which real operator to reach for before it ever calls list_operators cold.
  • datasets-and-samples.md: dataset lifecycle, views, fields and schema, sample and label edits, indexes, media import, workspaces.
  • analysis-and-labeling.md: brain methods, embeddings, model evaluation, data quality, auto-labeling, agentic labeling, annotation, zoo models, workflows, label schemas.
  • delegated-operations.md: running things in the background, checking on them, delegated run metadata.
  • agent-tools.md: the Voxel51 Agent's own operators, conversations, code execution, plugin and skill authoring, memory.
agent-tools.md covers a different kind of operator entirely: the ones behind the in-app Voxel51 Agent's own chat panel, including write_plugin and write_skill, operators that let an agent draft a new plugin or a new skill from inside a conversation. The same file is also precise about a real boundary: the in-app Agent's own internal memory and its own skill invocation mechanism are hardcoded into its own loop, not operators, and nothing reachable through this gateway touches either one.

Cloud or self-hosted, and more than one deployment at a time

The gateway address has one fixed shape, https://<host>/api/proxy/fiftyone-teams/mcp, and only the host changes. On Voxel51 Cloud that host is always app.voxel51.com. On a self-hosted deployment it is whatever your admin's install is reachable at. The manual setup step in the documentation spells out both cases side by side rather than assuming cloud:
DeploymentGateway URL
Voxel51 Cloudhttps://app.voxel51.com/api/proxy/fiftyone-teams/mcp
Self-hosted Voxel51https://<your-deployment>/api/proxy/fiftyone-teams/mcp
Gateway URLs for Voxel51 Cloud and self-hosted Voxel51 deployments.
One note if you are self-hosted or on-premises specifically: remote agent access ships behind a feature flag, off by default. If it is not already turned on for your deployment, contact your Voxel51 customer success contact to get it enabled before any of the steps above will work.
Registering it by hand, the same claude mcp add command shown above, names the server yourself, which is what makes more than one deployment straightforward: run that command again with a different name and a different host, and your agent has both available side by side.
A team with a staging deployment and a production one, or a consultant who supports several customers' self-hosted installs, adds a second entry the same way under a second name, for example voxel51-agent-gateway-prod next to voxel51-agent-gateway-staging, each pointed at its own host. The agent then asks which one by name instead of needing to be reconfigured every time you switch. The client id stays the same fixed string, voxel51-agent-mcp, for every deployment, because this auth server does not support a client registering itself automatically, every official per-client config, Claude Code, Claude Desktop, Cursor, VS Code, and GitHub Copilot, pins that same id rather than letting the client invent one.

From here, your agent is just another way into Voxel51

Follow the chain so far: you typed one message, your agent read a skill or a config entry, and from there it reached straight into the same platform you would otherwise open in a browser, the same datasets, the same panels, the same permissions. Voxel51 Skills and the Voxel51 MCP gateway are two doors into the same four things underneath: search and visualize, curate and evaluate, annotate, and automate.
The shape of it end to end: an AI assistant, through Voxel51 MCP or a Voxel51 Skill, reaching the same search, curate, annotate, and automate workflows the Voxel51 App itself is built on.
If you want to follow along on your own data instead of just reading about ours, Voxel51 Cloud is free to try at app.voxel51.com, nothing to self-host to get started. Before you even bring in your own imagery, the Explore tab is worth a look on its own: real, curated datasets across robotics, manufacturing, healthcare, autonomous vehicles, and more, each one a click away from landing in your own workspace.
Voxel51 Cloud's Explore tab, curated datasets organized by industry, ready to bring into your own workspace in one click.
The three examples below are what that same chain does once you are actually working, run against two of our own real datasets on a live deployment, two from Cursor in one session, the third from Claude Code's terminal in another. Nothing here is staged for this post, and nothing stops you from pointing the same prompts at a dataset from your own Explore tab.

First: what is in this dataset, and how big

With the gateway connected in Cursor, we asked for exactly what a new dataset deserves before anything else: its field schema and its size.
The Cursor prompt, restricted to the gateway's own tools only: list the available datasets, then scan VisA Anomaly Detection for data quality issues.
get_field_schema returned the full schema cleanly, about 70 fields once you count nested classification, detection, and segmentation subfields. The sample count was a different story. There is no dedicated count operator in this gateway's catalog at all, something the skill's own reference files say plainly rather than leaving you to discover it the hard way: no plain count, distinct values, or histogram call exists yet, only execute_code as a fallback for that specific question. We got the number a different way, reading dataset_size back out of a data quality panel call: 2,162 samples.
We also hit a real bug worth naming. get_operator_schema, the tool meant to tell you an operator's exact arguments before you call it, failed on every single operator we tried, including trivial ones such as list_datasets, always with the same error, a schema the tool did not actually return. The workaround was calling execute_operator directly with no arguments and reading the shape of what came back. If your agent hits the same error, that is the gateway, not your setup.

Second: a real data quality scan, with a threshold that lies about its own dataset

Next we asked the agent to scan VisA Anomaly Detection for blurry images, near duplicates, and anything else notable, using the real data quality operators rather than summarizing from memory.
CheckFlaggedWhat it means here
Blurriness424 / 2,162 (19.6%)A real issue, worth a manual pass before training
Near duplicates2,134 / 2,162 (98.7%)A threshold artifact, not a cleanup signal, see below
Exact duplicates0None found
Aspect ratio outliers0None found
Overexposed / blown out0None found
Blank / low texture0None found
Data quality scan results for the 2,162-sample VisA Anomaly Detection dataset, with the number of samples each check flagged and what each result means.
The blurriness number is real signal: 19.6% of samples fall under the panel's default threshold, in a range that runs from 8.4 up to 2398.5, heavily skewed toward the blurry end. The near-duplicate number is almost the opposite kind of finding. At the panel's default cosine distance threshold, 98.7% of the dataset gets flagged, which sounds alarming until you remember what VisA actually is: an industrial part inspection dataset, the same part photographed repeatedly under nearly identical lighting. The default threshold answers a question this dataset was never going to answer well. Finding genuinely redundant pairs here needs a tighter cutoff than the one the panel starts you on, not a cleanup pass on the headline number.
Both runs here are Agentic Labeling and Data Quality panel operators, see https://docs.voxel51.com/enterprise/agentic_labeling.html, the same panels a human would click through in the App, just reached through the gateway instead.

Every one of those calls lands in the dataset's own audit trail

This is the part that is easy to miss from a terminal transcript alone: open the same dataset's Activity tab in the App, and Adonai Vera updated 2162 samples via Compute aspect ratio (aspect_ratio, 2162 changes).
VisA Anomaly Detection's own Activity tab. An aspect ratio computation the agent triggered shows up attributed to the signed-in human, not a shared bot account.
That attribution is not cosmetic. The gateway authenticates as the approving user and nothing more, so list_operators only ever shows operators that user's role and dataset permissions currently allow, re-checked on every call rather than cached at connection time. Revoke the pairing, or have an admin remove a permission, and the next call reflects that immediately, the same live check a person clicking through the App would hit.
That live, per-user permission model is the same one documented for roles and dataset-level access generally, see https://docs.voxel51.com/enterprise/roles_and_permissions.html, the gateway does not introduce a separate permission system of its own.

Third: an agentic labeling run, including the moment it got the dataset's name wrong

The last test was the one example from the original prompt list we could run for real end to end: kick off agentic labeling on 20 unlabeled samples and report back when it finishes. We asked for it in Claude Code's terminal, against PhenoBench, a real plant phenotyping dataset already on the deployment.
We typed the dataset name the way people actually type dataset names, lowercase, "phenobench". It did not match. Rather than fail outright or guess, the agent queried the real dataset list, found the actual name, PhenoBench, capitalized differently than typed, and retried against that. Small moment, but it is the live permission and live catalog model from the sections above doing its job: the agent checks what is actually there instead of assuming.
The self-correction in progress: a call against "phenobench" finds nothing, a dataset list call finds the real name, PhenoBench, and the agent retries against it.
From there the run completed cleanly: 20/20 samples labeled, 0 parse failures, 0 inference failures, using an existing "Crop vs weed" labeling agent already configured on the dataset and the gpt-6.1-sol-responses model, in about 4 minutes.
The finished run, reported back in the same terminal session that typed the original request.
Everything the terminal reported is checkable in the App itself. The run detail panel shows the same numbers plus ones the terminal summary left out: 114,659 tokens across 20 requests, 77,346 of the input tokens served from cache, about 5,733 tokens per labeled sample.
The completed run's own detail view: dataset PhenoBench, label field al_plants, token usage broken out by input, cache, and output.
The labeling agent's own configuration, a crop versus weed prompt plus three annotated visual examples, exactly what the operator call reused rather than inventing new instructions for the run.
The labeled samples themselves render the same way any other detections field does in the App, crop in one color, weed in another, laid out across the sugar beet rows the dataset was photographed over.
PhenoBench's sample grid with the Agentic Labeling panel open beside it, the completed run listed with its token cost.
The 20 labeled samples themselves, crop and weed boxes drawn directly from the al_plants field the run wrote.
Long-running operators like this one can run in the background rather than block your agent's reply, see https://docs.voxel51.com/api/fiftyone.operators.delegated.html, which is what delegate=True and the two delegated operation tools are for.

Building your own: the shape that generalizes past this one skill

Nothing about this setup is specific to agentic labeling. The pattern is a skill file an agent fetches and follows, backed by a small, fixed tool surface that stays honest about what your account can currently do. Writing a new skill for your own team's workflow means writing a markdown file that names the real operator URIs for that workflow, the arguments worth knowing up front, and the failure modes a user will actually hit, the same shape as voxel51-agent-setup's own reference files for datasets, analysis and labeling, and delegated operations.
The pattern for writing one is documented directly: https://docs.voxel51.com/agents/developing_skills.html

Where this matters outside a demo org

  • Regulated or audited data access: teams in healthcare, finance, or defense contracting often cannot accept a shared service account touching patient, financial, or classified imagery, every access has to trace to a named person. A gateway that authenticates as the approving user and writes every action to that dataset's own activity history is the difference between an auditable agent and one compliance simply will not approve.
  • Self-hosted and air-gapped deployments: manufacturing, defense, and government teams frequently run Voxel51 on infrastructure with no path to a public SaaS endpoint at all. The gateway's only requirement is a host your agent can reach and an admin willing to turn the feature flag on, the same flow as the cloud case with one different address.
  • Multi-environment pipelines: an ML platform team running separate staging and production Voxel51 deployments, or a consultancy supporting several self-hosted customers, registers each as its own named gateway and tells its agent which one to use, rather than re-pointing a single config file every time the target changes.

Try it

The measurements and transcripts behind every number in this post are in measurements.json in this post's folder, next to the raw transcripts from both sessions.
How to connect an agent to your own Voxel51 deployment: https://docs.voxel51.com/agents/using_agents.html
To see PhenoBench the way our agent left it, open https://app.voxel51.com/dataset/phenobench/samples, the same dataset this post's agentic labeling run wrote al_plants onto.

Frequently asked questions

Headshot of Adonai Vera
Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera

Try Voxel51 for yourself

Related posts

Boost your visual and physical AI workflows.
View all