What Does It Actually Cost to Run an AI Agent? Claude, GPT, and Gemini, Head to Head
Sep 22, 2026
• 6 min read
Author
Adonai Vera
Adonai Vera is a Machine Learning Engineer & DevRel at Voxel51 with over 7 years of experience building computer vision and machine learning models using TensorFlow, Docker, and OpenCV. Adonai started as a software developer, moved into AI, led teams, and served as CTO. Today, he connect code and community to build open, production-ready AI — making technology simple, accessible, and reliable. LinkedIn | GitHub
See all articles by Adonai Vera
In this article
Key takeawaysWhy this mattersHow we measured AI agent costThe headline numbersAI agent cost by taskQuality and reliabilityModel-by-model summaryBroader task coverageHow each model fails matters as much as how it succeedsCaveatsThe Voxel51 Agent panel, in actionSo which model should you run?Questions? We have answers.
In this article
Key takeawaysWhy this mattersHow we measured AI agent costThe headline numbersAI agent cost by taskQuality and reliabilityModel-by-model summaryBroader task coverageHow each model fails matters as much as how it succeedsCaveatsThe Voxel51 Agent panel, in actionSo which model should you run?Questions? We have answers.
Talk to an AI expert
We wired the Voxel51 Agent up to Claude Sonnet 5, Claude Opus 5, GPT-5.1, and Gemini 3.1 Pro, sent every model the exact same prompts, tool schemas, and skills, and measured what each one actually consumed to get the job done. Not an estimate: the real per-request numbers each provider's own API reports.
Key takeaways
- Running the same three FiftyOne Agent tasks cost $0.109 on GPT-5.1 and $1.700 on Claude Opus 5, a roughly 15x spread on byte-for-byte identical prompts, tool schemas, and skills.
- Claude Sonnet 5 scored highest on quality, 8.67 out of 10 from an LLM judge, at $0.331, making it the best quality-per-dollar model in this run.
- All four models passed the rubric on all three core tasks, so judge score, not pass rate, is the signal that separates Claude, GPT-5.1, and Gemini 3.1 Pro.
- Worst-case cost varies far more than average cost: GPT-5.1's priciest single task was $0.52, while Claude Sonnet 5 hit $7.26 and Gemini 3.1 Pro hit $7.64 retrying against dead ends.
- Gemini 3.1 Pro consumed the fewest input tokens of the four models at 123,241, but token count alone did not predict final cost.
Why this matters
Model pricing pages tell you the cost per token. They don't tell you how many tokens the same task actually costs on each provider, and that number is not the same across providers, even when the prompt, tools, and available skills are byte-for-byte identical. We wanted a real answer, so we ran real conversations, with real tool calls, against a real FiftyOne dataset, and read the token counts straight out of the agent's own telemetry.
How we measured AI agent cost
We used voxel-agent-evals, an existing eval harness that drives the Voxel51 Agent in-process, with no mocking and no HTTP layer in between. A persona large language model (LLM) plays a real user across a multi-turn conversation, calling real Model Context Protocol (MCP) tools (dataset_summary, set_view, and others) and real Agent Skills. Every conversation ran against all four models, unmodified: same system prompt, same tool schemas, same skill index, same user message.
The headline numbers below come from three tasks that completed successfully on every model, which makes them a clean, apples-to-apples basis for comparing cost: describing an attached image (a pure vision task, no tools needed), summarizing a dataset in plain language, and turning a natural-language request into a Voxel51 view filter. Further down, we also show results from seven additional, more complex tasks (finding duplicates, checking data quality, visualizing embeddings, evaluating a model, importing data, and running a real object detection model), where results varied a lot more by model.
The headline numbers
Total tokens across the three core tasks, input vs. output:
What that actually cost, at each provider's published rate:
AI agent cost by task
Averages hide how cost shifts with the shape of the task. Broken out by scenario, you can see each model's cost profile on a simple vision task versus a tool-calling task.
Quality and reliability
Tokens and dollars are half the picture. We also scored every conversation with an LLM judge against the user's actual goal, and checked it against a rubric: did the agent call the right tools and land the right dataset state? All three core tasks passed the rubric on every model, so the judge score is the more useful signal for comparing quality here.
Model-by-model summary
| Model | Output tokens | Total cost | Average quality (1 to 10) | Pass rate |
|---|---|---|---|---|
| Claude Sonnet 5 | 1,560 | $0.331 | 8.67 | 100% |
| Claude Opus 5 | 3,363 | $1.700 | 8.00 | 100% |
| GPT-5.1 | 1,191 | $0.109 | 7.33 | 100% |
| Gemini 3.1 Pro | 2,564 | $0.277 | 7.67 | 100% |
Broader task coverage
The three tasks above are the clean comparison. We also ran seven more demanding tasks on every model: finding duplicate images, checking data quality, visualizing embeddings, evaluating a model's predictions, importing new data, running a multi-step analyze-and-filter workflow, and running a real YOLOv8n object detector. These are harder, more realistic tasks, and the results varied far more by model than the three clean tasks did.
Cost alone doesn't tell the full story on these harder tasks, since a cheap result isn't useful if the task didn't actually get done. The judge score below is the more honest signal here: it reflects whether the agent actually accomplished the user's goal, not just whether it made the right kind of tool call.
A few of these tasks (importing data, running the object detector, and the multi-step workflow) depend on operators that are designed to run through a live, connected Voxel51 App session. Our eval harness drives the agent headless, without a browser attached, so those specific operators can report success without the underlying action actually completing. That's a property of this evaluation setup, not a reflection of any model's ability, and it's part of why cost and judge score for those three tasks should be read as directional rather than conclusive.
How each model fails matters as much as how it succeeds
The harder tasks also surfaced a pattern that averages alone don't show: how a model behaves when it doesn't fully succeed. That behavior varied sharply, and it's arguably as important to a real deployment as the headline cost and quality numbers.
| Model | Behavior when it struggled |
|---|---|
| Claude Sonnet 5 | Retried extensively against a dead end |
| Claude Opus 5 | Consistently expensive, $2.62 to $5.96 per task, best of the four on analyze_filter_run_model`\ |
| GPT-5.1 | Recognized failure fast and stopped |
| Gemini 3.1 Pro | Retried extensively against a dead end |
- GPT-5.1 “fails cheap.” When a task didn't fully succeed, it recognized that fast and stopped. Its most expensive single task cost $0.52. It was also the most consistent performer relative to its cost.
- Claude Sonnet 5 and Gemini 3.1 Pro can “fail expensive.” Both had at least one task where they retried extensively against a dead end. Sonnet hit $7.26 on one task, Gemini hit $7.64 on another. Same model, wildly different cost depending on whether it happened to get stuck.
- Claude Opus 5 was consistently the priciest on these tasks ($3 to $6 per task) without a consistent quality edge to justify it, except one task (analyze_filter_run_model, judge score 8) where it was genuinely the best.
- Three of these tasks (data import, running a real model, and the multi-step workflow) are confounded by our headless test harness lacking a live browser, so treat those three as directional, not conclusive, about the models themselves.
Caveats
Anthropic's prompt-cache token accounting isn't broken out by the agent's current telemetry, so Claude's cost figures are a correct token count but a conservative upper bound on price: cached turns cost less in practice than shown here. Each scenario ran a single rollout per model, not an average over many, so treat this as a first real read rather than a statistically exhaustive one. All runs executed against an isolated sandbox dataset, not production traffic.
The Voxel51 Agent panel, in action
So which model should you run?
If cost per completed task is what you care about, GPT-5.1 is the value pick. It was the cheapest model in this run at $0.109, and it failed cheap when it failed at all. If you want the best answer on the first try, Claude Sonnet 5 scored highest with the judge at 8.67 while still landing under $0.35. Claude Opus 5 cost about five times what Sonnet cost for a lower judge score on these tasks, which is hard to justify outside the long, multi-step work where it pulled ahead.
The ranking isn't really the lesson though. The lesson is that identical prompts, identical tools, and identical skills produce very different bills depending on what's behind them, and that the worst case matters more than the average. A model that quietly burns $7 retrying a dead end is a different operational risk than one that stops and tells you it's stuck. If you're putting an agent in front of users, budget for the tail.
Models and prices move fast, so we'll rerun this. If there's a task you want in the next round, come tell us in the Voxel51 community.
Questions? We have answers.
On three identical FiftyOne Agent tasks in September 2026, total cost ranged from $0.109 on GPT-5.1 to $1.700 on Claude Opus 5. Claude Sonnet 5 came in at $0.331 and Gemini 3.1 Pro at $0.277. The same work, the same tools, and the same skills produced a roughly 15x spread in price depending on the model behind the agent.
GPT-5.1 was the cheapest in this run, at $0.109 across the three core tasks. It also had the lowest average judge score at 7.33, so the savings come with a small quality tradeoff on first-pass answers.
Not reliably. Claude Opus 5 cost about five times what Claude Sonnet 5 cost and scored lower with the judge on these tasks, 8.0 against 8.67. Price and quality only tracked each other at the bottom of the range.
Every conversation ran through voxel-agent-evals, an eval harness that drives the FiftyOne Agent in-process with no mocking and no HTTP layer. A persona model played a real user across multi-turn conversations calling real MCP tools and real Agent Skills, and the token counts came straight from each provider's own API.
Failure behavior. GPT-5.1 recognized dead ends quickly and stopped, capping its worst single task at $0.52. Claude Sonnet 5 and Gemini 3.1 Pro both retried extensively against dead ends, hitting $7.26 and $7.64 on a single task. Budget for the tail, not the average.
Adonai Vera
Adonai Vera is a Machine Learning Engineer & DevRel at Voxel51 with over 7 years of experience building computer vision and machine learning models using TensorFlow, Docker, and OpenCV. Adonai started as a software developer, moved into AI, led teams, and served as CTO. Today, he connect code and community to build open, production-ready AI — making technology simple, accessible, and reliable.
See all articles by Adonai Vera
Talk to an AI expert
Loading related posts...
© 2026 Voxel51 All Rights Reserved
