Tool Discovery at Scale: Cutting an Agent's Context by 97%

•
7 min read
Every call to "what can you do" handed our agent 21,777 tokens of tool definitions it mostly didn't need. A plain keyword search step cut that to under 700 tokens per call, a 96.7% reduction, with no measurable drop in whether the right tool got picked. This is the general shape of that problem, what we tried, and how we validated it end to end on the Voxel51 Agent.

Key takeaways

  • Model Context Protocol (MCP) tool discovery that returns every tool definition on every call gets expensive as a catalog grows. On the Voxel51 install measured here, list_operators returned 512 operators and panels across 40 plugin namespaces on every discovery call.
  • Adding a dependency-free, field-weighted keyword search step to list_operators cut the response from 21,777 to a mean of 711 tokens on the external MCP gateway (96.7%) and from 22,814 to a mean of 530 tokens on the Voxel51 Agent (97.7%).
  • In six realistic queries against the 512-operator Voxel51 catalog, the keyword scorer put the operator a human would pick first in five, and its top result for the sixth was also the right operator, so this MCP tool discovery fix needed no embeddings model.
  • Ranking for MCP tool discovery has to run after permission filtering, so a search query can never surface an operator that the caller isn't already allowed to see.

The problem: MCP tool discovery doesn't scale with your tool catalog

Any large language model (LLM) agent that discovers tools through a "list what's available" call eventually has to answer that question against a catalog that's grown past a demo. The easy answer is to hand over every tool definition, every time, regardless of what the current task actually needs. That's free at ten tools. It gets expensive in a way nobody notices until someone measures it, once a catalog reaches the hundreds.
We validated this on the Voxel51 Agent, our own Voxel51 agent, because it's the one we could instrument end to end down to real token counts. The problem and the fix aren't specific to it: a recent post on this blog covered the Voxel51 Agent Gateway, which lets a coding agent connect over OAuth and act as the signed-in user. That solved who's allowed to do something. It didn't touch what the agent can find in the first place, and that second question hits anything built the same way, any Model Context Protocol (MCP) server, any tool-calling agent, once its own catalog grows.
On the install this post measures, that catalog is 512 operators and panels across 40 plugin namespaces. That's not a worst case, it's what a plugin ecosystem that's been running for a while actually looks like once every installed operator counts, not a curated demo set. All 512 got dumped into the agent's context on every single discovery call, whether the task at hand was "find duplicates" or "export to COCO."

What we explored, and what the experiment said

There were two shapes the fix could take: a dependency-free keyword scorer, or an embeddings-based semantic ranker. We built the cheap one first and tested whether it was good enough before reaching for the expensive one.
It was. Across six realistic queries run against the real 512-operator catalog, the keyword scorer put the operator a human would pick in the #1 slot five times out of six. The one "miss" turned out to be correct too, by a strict check of the operator's literal name, it just didn't spell the query term in that field, only in its description. That's the order worth keeping: build the cheap version, measure it against your own catalog, and only reach for an embeddings model once keyword search demonstrably fails on real queries, not before.
That order also matches what's been published elsewhere. Anthropic's own Tool Search Tool reports an 85% token reduction and real accuracy gains just from adding a search step instead of a flat list, no embeddings required. A separate benchmark across 2,792 tools found keyword-only search picked the right tool 34% of the time, versus 94% for a hybrid semantic-plus-keyword approach, with keyword search carrying most of that lift on its own. We didn't need the literature to tell us to try keyword search first. Our own experiment already said it was enough. It's just reassuring when outside evidence and your own numbers agree.

What we built: one search step, reused everywhere an agent asks

Voxel51 operators reach the agent through two separate surfaces, and both had the identical flat-dump problem, for different reasons:
  • The external MCP gateway. Used by Claude Desktop, Claude Code, Codex, and Antigravity. Its list_operators tool returned the full, permission-filtered catalog on every call, with no way to narrow it.
  • The Voxel51 Agent. Backed by a separate server, fiftyone-mcp-server, whose own list_operators tool has the exact same shape: one tool-call result holding every operator, every time.
The fix in both places is the same, dependency-free, field-weighted keyword scorer: match a query against each operator's name, label, plugin, and description, rank, cap at 10 results. No embeddings model, no new infrastructure, no new dependency. Name and label matches count for more than a plugin-name match, which counts for more than a description match, since description is the longest field and the most likely to collide by coincidence.
list_operators(query="find duplicate images")
# -> find_exact_duplicates, find_near_duplicates, ...
# instead of all 512 operators, every time
One detail mattered more than it looked like it would: permission ordering. The gateway already filters its operator list through an allowlist/denylist (mcp_enabled, plus a hardcoded list of destructive operators such as delete_dataset that can never be exposed even by mistake). Ranking has to run strictly after that filter, not before, so a query can never surface an operator a caller wasn't already permitted to see. It's an easy thing to get backwards if you bolt search on as an afterthought instead of writing it against the existing filtered list.

What we measured

Same real operator catalog (512 operators, 40 plugin namespaces), same dependency-free scorer, measured on both surfaces.
SurfaceBefore (full list)After (mean/query)Reduction
MCP gateway21,777 tokens711 tokens96.7%
Voxel51 Agent22,814 tokens530 tokens97.7%
list_operators response size in tokens before and after keyword search, on the external MCP gateway and the Voxel51 Agent.
Six hand-picked, realistic queries ("find duplicate images," "export to COCO format," "evaluate my model's predictions," "visualize embeddings in 2D," "run an agentic labeling job," "compute image quality metrics"), checked against this install's real operator names, not invented ones. Five of six put the operator a human would pick in the #1 slot, by a strict check of the result's literal name. The sixth ("compute image quality metrics") ranked dq_scan_compute first, which is in fact the right operator. It simply doesn't spell "quality" in its own name, only in its description, so a strict name-substring check counts it as a miss even though the scorer got it right. n=1 machine, one real accumulated catalog, not a curated benchmark set, stated here because six queries against one catalog is a small sample and we're not pretending otherwise.
The ranking step itself runs in 2.67ms on the gateway, pure in-memory scoring over an already-fetched list. The Voxel51 Agent's number, 327.2ms, looks slower, but that's not the scorer: it's Voxel51's own operator-registry scan running underneath it on every call, present whether or not a query is attached. The new scoring step adds sub-millisecond overhead on top of a cost that was already there.

Building your own: the shape that generalizes

None of this needed an embeddings model, a vector index, or new infrastructure. This generalizes past Voxel51 entirely, to any agent with a tool catalog that grows: the shape is small enough to hold in your head.
  • Score, don't eval. Case-insensitive substring and whole-token matching across a handful of weighted fields. Name and label weigh more than a plugin or description match.
  • Filter first, rank second, always. Whatever already decides who's allowed to see a result must run before ranking touches it, on every code path, including the no-query and empty-query paths.
  • Cap untrusted input before you tokenize it, not after. The catalog you're searching is yours and bounded. The query string is the caller's and isn't. Treat them differently from the start.
  • Verify your own premise about what's actually being deferred. "The tool list is too long" and "the tool list contains one operator per schema" are different claims. Check which one is actually true for the specific registry in front of you before building around it.

Where this matters beyond a single install

This is dev tooling, not a dataset finding, so the stakes are about where any agent-plus-plugin ecosystem grows past a demo. Three real shapes this shows up in:
  • Any deployment with a mature plugin ecosystem. The 512-operator, 40-namespace catalog measured here isn't a worst case: it's what a plugin ecosystem that's been running for a while looks like once every installed operator counts, not a curated demo set. A production deployment with years of accumulated custom operators is the same shape, just bigger.
  • Any MCP server whose tool catalog is actually dynamic. Plugin marketplaces, internal tool registries, and anything where "what's available" changes as teams ship more of it, as opposed to a fixed API surface that only changes on a release cycle.
  • Any agent platform billed per token. A 96.7-97.7% reduction in a payload that ships on every single discovery call is a cost line, not just a latency one, at any scale where "every single call" adds up.

Try it

The agent gateway and the skill that configures it: https://docs.voxel51.com/agents/using_agents.html
The Voxel51 operators API this all sits on top of: https://docs.voxel51.com/api/fiftyone.operators.html

Frequently asked questions

Headshot of Adonai Vera
Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera

Try Voxel51 for yourself

Related posts

Boost your visual and physical AI workflows.
View all
by Adonai Vera, DevRel
AI agents, Product and news
•
by Adonai Vera, DevRel
Data annotation, Product and news
•