Voxel51 Agent: Reduce Manual Data Work Across Model Development
Author
Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera
Marketing Manager
Voxel51 helps ML and physical Al teams improve model performance with scalable visual data analysis, curation, and evaluation. Our customers include Bosch, Porsche, Ford, Sony, LG, Walmart, Honda, and the US Gov; while researchers at Google, Microsoft, and NVIDIA use Voxel51 to build state-of-the-art physical Al datasets.
See all articles by Layla Yun
In this article
Try Voxel51 for yourself
ML teams spend a significant part of model development working through the data. A seemingly simple question like “why is pedestrian recall dropping at night?” or “why does grasp success rate drop for reflective objects?” can turn into a chain of work: isolate the failures, inspect samples or episodes, compare predictions and ground truth, search for similar samples, check dataset coverage, curate a new subset, rerun evaluation, and decide what to change next.
That work is critical to improving model performance. However, much of it still requires engineers to spend hours manually querying datasets, writing code, inspecting results, and deciding what operation comes next.
Voxel51 Agent is built on top of the broader Voxel51 platform, so it can move across data exploration, curation, annotation, and model evaluation rather than being limited to one part of the workflow. Start with the question or outcome you care about in natural language, and the Agent can reason over your datasets and model evaluations, execute Voxel51 operations, write custom code, and carry the work forward into curation, evaluation, annotation, and other actions. When a process becomes repeatable, it can also be captured as a reusable skill or extended with custom plugins.
Voxel51 Agent handles more of the data work required to investigate and iterate, giving engineers more room to focus on what the model needs next. Voxel51 Agent is available now, so you can try these workflows for free to get started.
Key takeaways
- Work through data faster: Start with a question or outcome you care about, then let the Voxel51 Agent handle the querying, analysis, and operations required to get there.
- Go from investigation to action: Voxel51 Agent turns what you find directly into curation, evaluation, annotation, or custom analysis.
- Automate what you repeat: Capture recurring data and model development processes as reusable Voxel51 Agent skills that can be applied across datasets and model versions.
- Extend the Agent to your workflow: Use custom SDK code and plugins for proprietary or one-off work, and bring Voxel51 capabilities into compatible AI tools through MCP.
What you can do with Voxel51 Agent
The value of finding a failure mode is what you can do with it next. Voxel51 Agent can move from investigation into curation, evaluation, annotation, or custom analysis without requiring you to manually execute every operation along the way.
Find failure modes and coverage gaps in natural language
Large physical AI datasets can bury a narrow failure mode inside millions of samples. Voxel51 Agent lets you start with the problem you want to investigate, such as:
- Show me episodes similar to these failed grasps
- Find high-confidence false positives for cyclists
- Find underrepresented scenarios in this dataset
Natural language search for rare lighting scenarios in Voxel51
The Agent queries samples, analyzes labels and predictions, computes and searches embeddings, creates dataset views, and runs evaluations using the same underlying capabilities available in Voxel51.
From there, the resulting samples can become a targeted evaluation cohort, a more representative training subset, or a queue of data that needs annotation.
Automate recurring curation and evaluation workflows
The State of Visual and Physical AI 2026 found that a third of teams (34%) spend more than half their time on exploration, curation, cleaning, and quality checks. Much of that work still involves reconstructing views, running commands, inspecting results, and deciding what should happen next.
Voxel51 Agent can turn those sequences into an automated workflow. It can inspect the dataset, run the relevant operations, surface results for review, and continue into the next step.
That makes it useful for recurring workflows like:
- Auditing incoming data for duplicates, outliers, or quality issues
- Investigating a model regression and building a failure cohort
- Identifying high-value samples and preparing them for annotation
- Comparing a new model against critical evaluation slices
- Rerunning a standard dataset or release-readiness check
Voxel51 Agent computes blurriness, brightness, and entropy operators to automatically flag low-quality samples.
The Agent surfaces the result of each step for review before continuing, so engineers can make the decisions that matter without manually executing every operation in between.
For workflows you repeat often, the process can be captured as a reusable skill so the Agent can apply the same steps to new datasets or model versions.
Bring annotation into the loop
Annotation is rarely as simple as "send this dataset to labeling." Teams first need to decide what is worth labeling, generate or refine annotations, review quality, and feed the resulting data back into model development—often moving between different tools along the way.
With Voxel51 Agent, those steps can remain part of the same data workflow. Combined with Voxel51's Annotation capabilities and Agentic Labeling, the Agent can move from identifying valuable samples to generating labels, reviewing results, and preparing the data for the next model iteration.
With Voxel51 Agent, you can easily identify annotation errors and set up HITL review workflow
For example, the Agentic Labeling Skill shows one way to package automated labeling, validation, and human review into a repeatable process the Agent can follow.
Extend Voxel51 Agent to fit your workflow
| Extension | What it adds | Best for | Reusable? |
|---|---|---|---|
| Skill | How to combine existing capabilities into a workflow | Team processes you repeat across datasets and model versions | Yes, shareable across the team |
| Plugin | A new capability (operator) the Agent can call | Proprietary services, custom formats, in-house metrics | Yes, private or shared |
| Custom SDK code | One-off logic generated in the conversation | Long-tail analysis that does not fit existing operations | Not by default, can seed a skill or plugin |
Encode how your team works with agent skills
A Skill teaches the Agent how your team carries out a particular workflow, including the steps, tools, criteria, and decision points involved.
For example, a tool might allow the Agent to compute embeddings. A skill can define how your team uses those embeddings to audit incoming data for redundancy, coverage gaps, and outliers—and what to do with the results.
Once encoded as a skill, that process can be reused across datasets and model versions and shared with the rest of the team instead of living in a notebook, internal document, or someone’s head.
Browse the Agent Ecosystem to see what the Agent can do out of the box, or build your own workflow.
Add new capabilities with plugins
Using Voxel51 Agent to create a custom plugin for flagging low-confidence predictions.
A generic agent that only knows public APIs stops being useful the moment your workflow touches something proprietary—an internal inference service, an in-house quality metric, or a sensor format nobody outside your company has heard of.
Plugins add those capabilities to Voxel51 Agent. Any operator registered through Voxel51 Plugins can become available to the Agent, whether it connects an internal service, supports a custom data format, exposes a proprietary metric, or adds a domain-specific operation.
The Agent can also help generate plugin code from the conversation. You can inspect the implementation, test and refine it, and then make the capability available for reuse. Plugins can be kept private or shared across your team.
Write custom code with Voxel51 Agent
For one-off work that falls outside existing operations, Voxel51 Agent can generate SDK code directly in the conversation.
This gives the Agent a way to handle the long tail of ML development without leaving the workflow to build a separate script or notebook first. And if that one-off work becomes useful repeatedly, it can become the starting point for a reusable plugin or skill.
Built on MCP for your models, infrastructure, and AI tools
Extensibility also applies to where the Agent runs and which AI tools your team uses. Voxel51 uses the Model Context Protocol (MCP), an open standard for connecting AI applications with external tools. That means the same capabilities available to Voxel51 Agent can also be made available to compatible assistants and developer tools in your existing stack. This gives developers flexibility to:
- Bring your preferred model: Connect to over 100 providers (including Anthropic, OpenAI, and Google) or your own self-hosted model.
- Run where your data runs: Available across all Voxel51 deployment options—cloud, hybrid, on-premises, air-gapped, and managed.
- Use Voxel51 from compatible AI tools: Expose Voxel51 operations through MCP instead of limiting them to one agent interface.
Model choice is essential to giving teams more control over operating costs. We benchmarked the same Voxel51 Agent tasks across Claude Sonnet 5, Claude Opus 5, GPT-5.1, and Gemini 3.1 Pro and found cost differences of up to 15x.
The more useful finding was how models behave when a task doesn’t go smoothly: some recognize a dead end and stop, while others repeatedly retry unsuccessful operations. How a model uses tools, retries operations, and handles failures can have a much bigger effect on what an agent actually costs to run.
Being able to swap models means you can match the model to the task—and to your budget.
From one-off analysis to continuous ML development
ML development will always require engineers to decide what matters: which failures are important, where the data is weak, and whether a model is ready. What doesn’t need to remain manual is all the work required to carry those decisions from one step to the next.
Voxel51 Agent changes how much of the work underneath those decisions has to be executed manually. Engineers can start with the question or outcome they care about, let the Agent work through the relevant data and model operations, and step in where judgment is required.
You can get started with Voxel51 Agent for free and explore these workflows directly.
FAQ
Voxel51 Agent is an AI-powered assistant for Voxel51 that can use Voxel51's data, model evaluation, annotation, and extensibility capabilities from natural language instructions. It can investigate datasets and model behavior, carry out Voxel51 operations, and continue into actions like curation, evaluation, annotation, and custom analysis.
A chatbot primarily answers questions. Voxel51 Agent can take action on your Voxel51 environment using tools: querying and curating datasets, creating views, using embeddings, running evaluations, working with annotation, and executing specialized workflows through skills and plugins.
Voxel51 Skills are a set of instructions that teach an agent how to carry out a particular workflow using available tools. Skills are useful for encoding repeatable team processes: not simply what operations exist, but how your team expects them to be used.
A plugin extends what the Voxel51 Agent can do. A skill teaches the Agent how to combine available capabilities to accomplish a task. A plugin can introduce a new operation, while a skill can define how and when the Agent should use it.
No. Pipeline orchestrators execute predefined jobs and dependencies. Voxel51 Agent addresses a different part of the ML lifecycle: the iterative data and model development work where the next action often depends on what you discover. The two can work alongside each other.
The Voxel51 MCP Server exposes Voxel51's capabilities to compatible AI assistants and developer tools, so you can bring Voxel51 into the AI environment you already use rather than being limited to a single agent interface.
Voxel51 Agent works with over 100 providers, and the right choice depends on your workload. We benchmarked identical agent conversations across Claude Sonnet 5, Claude Opus 5, GPT-5.1, and Gemini 3.1 Pro to measure real token consumption, cost, and output quality per task.
Machine Learning Engineer
I'm exploring how we can make agents see, understand, and transform the world around us. I’m dedicated to building systems that bridge perception and action, shaping a future where technology seamlessly integrates into our lives.
See all articles by Adonai Vera
Marketing Manager
Voxel51 helps ML and physical Al teams improve model performance with scalable visual data analysis, curation, and evaluation. Our customers include Bosch, Porsche, Ford, Sony, LG, Walmart, Honda, and the US Gov; while researchers at Google, Microsoft, and NVIDIA use Voxel51 to build state-of-the-art physical Al datasets.
See all articles by Layla Yun
Try Voxel51 for yourself
Loading related posts...
© 2026 Voxel51 All Rights Reserved
