Gemma 4 + FiftyOne Agent: Bring Your Own Open Source Model
Jul 21, 2026
• 9 min read
In this article
Key TakeawaysGetting an open model running on our own machineWhat it actually took to fit it in memoryWiring it into FiftyOneAny provider, hosted or your own modelWatching Gemma actually thinkTracking what it costs, built with FiftyOne's own toolsWhere it actually hit a wallWhy this matters for physical AIIf you want to try itFrequently asked questions
In this article
Key TakeawaysGetting an open model running on our own machineWhat it actually took to fit it in memoryWiring it into FiftyOneAny provider, hosted or your own modelWatching Gemma actually thinkTracking what it costs, built with FiftyOne's own toolsWhere it actually hit a wallWhy this matters for physical AIIf you want to try itFrequently asked questions
Talk to an AI expert
FiftyOne is built to be the harness for physical AI data, and part of what makes it that harness is the freedom to connect any model you want. This is the open source half of that story.
FiftyOne Agent is not built around one brain. Most agent tools give you whoever trained the model behind them, and that is what you get every time you open the app. FiftyOne Agent does not work that way, on purpose.
The part of FiftyOne Agent that makes this possible is small and plain on purpose. It talks to a model through a standard endpoint, and as long as that endpoint answers in the same shape most serving tools already use, FiftyOne treats it like any other model, sitting in the same list as Claude and Gemini. That includes a model you run yourself, on your own computer, with weights you downloaded and can inspect line by line. That flexibility is the whole point of the harness. This post is about one specific way to use it, an open source model, hosted by nobody but us.
So we tried it for real. We picked an open model, Google's Gemma 4, served it on our own hardware with vLLM, wired it into FiftyOne Agent, then built a small tool of our own, using FiftyOne's own plugin framework, to watch it work. Along the way we hit real limits, some in our own hardware, some in the tools themselves, and we are putting all of it in here, not just the parts that make a clean demo.
Key Takeaways
- FiftyOne Agent connects to any model behind a standard chat endpoint, so a model you run yourself appears in the same picker as Claude and Gemini, added and used the same way.
- We served Google's open source Gemma 4 with vLLM on a single RTX 4070 Ti and connected it through one Add provider form, with no adapter or custom code.
- The real footprint of a model is larger than its download size. Gemma 4 E4B downloaded at about 4 GB but loaded to roughly 10 GB, so we moved to the smaller E2B to leave room for tool calling and a full conversation.
- Running your own model inherits the plumbing a hosted model hides: GPU memory limits and tool-layer guardrails, like FiftyOne refusing a
distinctresult over 2.5 million characters. - For physical AI data that cannot leave your infrastructure, a self-hosted agent lets the footage and the model reasoning about it stay in the same room.
Getting an open model running on our own machine
We picked Gemma 4, Google's open model family. It ships under an Apache license, comes in a few sizes, and can call tools through its own function calling protocol, which matters once an agent has to do more than chat.
To actually run it, we used vLLM, an open source engine with a ready made recipe for Gemma 4 already written up. One command gave us a server on our own GPU that spoke the same standard chat format most serving tools already use, the exact shape FiftyOne Agent already expected. No translation layer, no custom code on our side to make the two talk to each other.
What it actually took to fit it in memory
| Decision point | What we did |
|---|---|
| Model size | Moved down to the smaller E2B |
| Tool calling | Kept it on, with room freed by E2B |
| Conversation vs. concurrency | Chose a long conversation over serving multiple people |
| Memory crashes | Set the allocator to manage space more flexibly |
This is the part a lot of write ups skip, and it is worth being honest about, since model size is not just a number on a page once you are the one running the thing.
Our GPU, a single RTX 4070 Ti, has twelve gigabytes of memory. Gemma 4 comes in a few sizes, and we first tried the larger one, called E4B. Its download was only about four gigabytes, which looked easy to fit. Once it actually loaded onto the GPU though, its real footprint turned out to be closer to ten gigabytes, since the download size and the loaded size are not the same thing at all. That left almost nothing free for an ongoing conversation.
It got tighter from there. Getting tool calling to work required a flag that tells the server to actually parse tool calls, and turning that on took roughly another gigabyte during loading, on top of everything else. Between the larger model and that flag, we kept running out of memory before a single conversation even started.
We fixed it by moving down to the smaller Gemma 4 size, called E2B, which left enough room for tool calling and a real conversation at the same time. We also had to choose between a longer conversation and serving more than one person at once, since both draw from the same pool of memory, and we chose a long conversation over concurrency, since this is a tool built for one person at a time. One more setting, telling the memory allocator to manage space more flexibly, cleared up the last of the crashes. None of this shows up in a marketing page for an open model. All of it shows up the moment you actually try to run one.
Wiring it into FiftyOne
Once our server was actually up and stable, hooking it up to FiftyOne took one form. Under Agent Settings, in Connections, we chose Add provider, named it Gemma, and set the provider type to the standard option, since that is the shape our server speaks. The Endpoint field got the address of our own machine, and Custom model names got the exact id our server was serving. FiftyOne's enterprise docs walk through this same screen at docs.voxel51.com/enterprise/agent.
That was it. No adapter, no special case written for Gemma anywhere in the code. FiftyOne's harness already spoke the one protocol our own server spoke too.
Any provider, hosted or your own model
This is the part worth sitting with. Once we saved that form, our model showed up in the exact same picker as every hosted model, grouped under its own name, no different section, no separate flow. The same settings screen that connects FiftyOne to Claude or Gemini connects it to a model running in the next room.
FiftyOne does not treat a state of the art hosted model and a model you run yourself as two different categories of thing. They are just entries in the same list, added the same way, picked the same way, and used the same way once picked. Whatever reason you have for wanting your own model in the loop, cost, privacy, control, or just curiosity, the path to get it there is not a special case. It is the same form everyone else already uses.
That matters most for the kind of data FiftyOne is built around. Footage from a factory floor, a hospital room, a self driving car, often cannot go to a public API at all. If the agent looking at that footage has to be as private as the footage itself, it has to run in the same room, and now it can.
Watching Gemma actually think
We opened a real dataset, coral reef images with ground truth labels and model predictions, and just asked Gemma to describe it. It answered honestly about its own plan before doing anything: “To give you a description, I first need to load the dataset. What is the name of the dataset you would like me to describe.”
Once we confirmed, it pulled the dataset's basic context, then reached for a tool called
fiftyone-data-distribution, treating a skill name as if it were something it could call directly. That call failed. What we liked was what happened next. Gemma noticed, said so out loud, “since the previous attempt to call fiftyone-data-distribution as a direct tool failed, I will now try to use the general dataset_summary function,” and asked before trying again. We said yes, and it came back with an accurate summary, media type, sample count, tags, splits, and the most common labels in the dataset.A little later we asked it to filter for a label that was not actually in the dataset under that name. The filter came back empty, and instead of guessing again, Gemma explained itself and asked which field to check. We told it to look at
ground_truth instead, and it called distinct to find the real label names. That is a normal, healthy back and forth, and it happened without a single hosted model in the loop.Tracking what it costs, built with FiftyOne's own tools
Once you are the one paying for the electricity and the GPU, you start wanting to know what it is actually using. So we built a small FiftyOne plugin of our own to keep an eye on our own model while it worked, and the way we built it is its own part of this story.
We used FiftyOne's plugin framework, documented at docs.voxel51.com/plugins, the exact same system anyone building inside FiftyOne reaches for. The plugin is a hybrid panel, a small Python backend paired with a React frontend, polling vLLM's own metrics endpoint every few seconds. We did not add a logging database or any custom instrumentation, we just read the numbers vLLM already tracks: how many requests it has handled, how many are waiting right now, how much of the GPU's cache is in use, and how long replies are taking, plus two small live charts for throughput and time to first token.
We did not just write the code and hope it worked either. We drove the actual, running FiftyOne App with Playwright, an open source browser automation tool, opened the panel for real, clicked the real refresh button, and watched the real numbers move before we trusted any of it. The same plugin framework that lets you build almost anything inside FiftyOne is also what let us build a tool for watching our own agent, and the same kind of browser automation anyone can reach for is what let us prove it actually worked, not just that it compiled.
Watching those numbers turned something invisible into something we could just look at, right where we were already working.
Where it actually hit a wall
Two different limits showed up fast, and neither one was really about Gemma 4's own context window, even though that is the first thing we suspected when a tool call failed.
The first was our own hardware, already covered above, the real cost of fitting a model, its tool calling support, and a long conversation into twelve gigabytes at the same time. That is a real tradeoff you take on the moment you serve a model yourself instead of renting someone else's, and no amount of clever configuration makes a fixed amount of memory bigger.
The second limit lived inside a tool, not the model. When Gemma called
distinct on our ground truth field, the real answer was huge, over two and a half million characters of raw text. FiftyOne's own tool layer caught that before it ever reached the model and sent back a plain refusal instead: “Tool distinct response too large, use more specific parameters to reduce the response size.” That is not a bug. No model, hosted or self served, should be handed several megabytes of raw output in one go, and it is exactly the kind of guardrail you want once real tools start touching real data.Put together, both limits taught us the same lesson. Running your own model does not just hand you a new brain, it hands you all the plumbing decisions that a hosted model normally hides from you. That is the price of the control you get in return.
Why this matters for physical AI
FiftyOne already sits at the center of physical AI work, robotics, autonomous vehicles, factory inspection, anything where the raw material is images, video, point clouds, or sensor logs instead of plain text. That is the hardest kind of data to hand to a public API, and often the kind you are simply not allowed to.
Bringing your own model into that same room closes the loop. The footage stays where it already lives, and the model reasoning about that footage can live right there with it. You end up with an agent that can plan an annotation run or question a label distribution without a single frame ever leaving your own infrastructure.
If you want to try it
You do not need Gemma 4 specifically, any model you can serve behind a standard endpoint will work the same way. Serve it on your own hardware, vLLM is a solid place to start, and budget more memory than the download size alone suggests. Then in FiftyOne, open the Enterprise Agent view, go to Agent Settings, then Connections, and add a provider pointing at your own endpoint, the exact walkthrough for that screen lives at docs.voxel51.com/enterprise/agent. Pick your model from the picker like any other and start talking to it. And if you want to see what it is costing you, FiftyOne's plugin framework at docs.voxel51.com/plugins is where we would start, and the full agent harness lives at docs.voxel51.com/agents.
The best part of running your own agent is not that it is free. It is that you finally get to watch it think, mistakes, guardrails, and all.
Frequently asked questions
No. Any model you can serve behind a standard chat endpoint works the same way. Gemma 4 is just the open model we chose for this walkthrough.
In the Enterprise Agent view, go to Agent Settings, then Connections, and choose Add provider. Set the provider type to the standard option, point the Endpoint field at your own server, and enter the exact model id under Custom model names. The full walkthrough is at docs.voxel51.com/enterprise/agent.
More than the download size suggests. On a 12 GB RTX 4070 Ti, the larger E4B model loaded to about 10 GB, and enabling tool calling added roughly another gigabyte. Moving to the smaller E2B left enough room for tool calling and a long conversation at the same time.
Cost, privacy, control, or curiosity. It matters most for data that cannot go to a public API, like factory floor, hospital, or self-driving footage, where the agent can now run in the same environment as the data.
Two showed up: fixed GPU memory, which no configuration can expand, and tool-layer guardrails, such as FiftyOne refusing to pass a tool response over 2.5 million characters to the model. Both are tradeoffs you take on when you serve a model yourself.
Loading related posts...
© 2026 Voxel51 All Rights Reserved
