Book a demoTry Voxel51
Blog
Physical AI

When the LLM Becomes the Policy: A Primer on Agentic Robotics

by Mohamad Louai Shehab
Nov 10, 2026
•
14 min read
Author
Photo of Mohamad Louai Shehab
Mohamad Louai Shehab
Postdoctoral Researcher, University of Michigan Robotics
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab

Try Voxel51 for yourself

Get started for free

Key takeaways

  • Agentic robotics puts a multimodal large language model (LLM) in a robot’s control loop, where it calls tools, checks what happened, and picks the next call until the task is done.
  • In a September 2026 study, GPT-6 Astra averaged 22.5% success across RoboDojo’s 42 simulated tasks with no robot fine-tuning, enough to rank first against the leaderboard’s September 10 snapshot.
  • Agentic robotics systems fall into four families, sorted by what the LLM produces: motion commands, programs, calls to learned policies such as vision-language-action models (VLAs), or changes to training code.
  • Agentic robot policies are still tested mostly in simulation, and they remain weak on precision, safety, cost, and speed.
  • For robot data teams, agentic robotics shifts the work from collecting more demonstrations to curating tools, demos, and agent traces.

1. A robot policy without additional robot fine-tuning

You know that feeling when AI picks up a new buzzword every week? “Agents,” “harnesses,” “tool calls,” “orchestration”? Well, strap in. That whole vocabulary is now arriving in robotics :)
Here’s how. In September 2026, researchers gave OpenAI’s GPT-6 Astra [1] camera images and a single motion tool, move_eef, plus a give_up button. No robot fine-tuning. Then they ran it on all 42 of RoboDojo’s simulated tasks. It averaged 22.5% success. Measured against the leaderboard’s September 10 snapshot, it would have ranked first [2]. OpenAI’s own GPT-5.5, released about four months earlier, managed under 1%.
And the board keeps moving. As of early November, the top trained policy sits at 34.97%, while PyRUA-Lean, which has Astra write short programs instead of single moves, is at 50.35% on the agent track [3, 4]. RoboDojo and RoboMME now even list agent systems on their own tracks [3, 5].
Handing a large language model (LLM) control of the robot like this is called “agentic robotics,” and it might be the field’s newest buzzword. In June, at ICRA 2026, Ken Goldberg used his plenary talk to ask whether agentic coding can close the gap between robotics’ “two cultures”: careful, engineered automation and data-hungry robot learning [6]. His answer leans on LLMs writing robot code, one of the four families we cover in Section 4, and he’s an author on two systems we’ll meet there, ASPIRE and ENPIRE. This November, CoRL 2026 hosts a full workshop on agentic robotics, with speakers from Google’s Gemini Robotics team, NVIDIA, and Goldberg himself [7].
If some of this sounds alien, don’t worry. This piece will walk through what agentic robotics is, where it came from, how people build it, why it suddenly works, where it still breaks, and what it means for your data.
One note: when I say “LLM,” I mean a multimodal model that reads camera images as well as text. Some papers call these vision-language models (VLMs). Same idea.

2. So what is agentic robotics, actually?

Here’s the short version: the LLM stops handing off the work and starts doing it.
For the past few years, the standard recipe split the job in two. Say you ask a robot to clean a table. A language model looks at the scene and writes the plan: pick up the sponge, move above the spill, wipe, put the sponge back. Then a trained policy (usually a vision-language-action model, or VLA) or a classic motion planner carries out each step. The LLM was the manager. It never touched the motors. Researchers call this orchestrating robot policies [8].
But the LLM was already in a loop there, checking progress and picking the next step. So here’s the real difference. Agentic robotics means the LLM acts through tools, looks at what happened, and picks the next call, again and again, until the job’s done or it gives up. The question is how low those tools go. At one end, a tool is “wipe the spill,” and a trained policy figures out the motion. At the other, a tool is “move the gripper to these coordinates,” and nothing learned sits between the LLM and the motors. What’s new in 2026 is that the LLM can finally work at that second end.
Most diagrams draw a single arrow from “LLM” to “robot,” and you only see what’s in between when you open the code. So let’s open it up.
Figure 1. The agent loop, one level down. The LLM never talks to the robot directly. It calls a tool such as move_eef (move the end effector, that is, the gripper). The harness checks the call and plans a path, the robot API turns that into joint commands, and the harness reports back what happened.
In plain terms:
  • The agent is the whole loop: model, harness, and robot together.
  • Tools are the only actions the model is allowed to take. In the RoboDojo study’s harness, the main one is move_eef. The model says where the gripper should go (x, y, z, a rotation, open or closed), and everything else is handled for it.
  • The harness is ordinary code wrapped around those tools. It checks every call, plans the motion, and decides what the model sees next.
  • The API is the robot’s own interface.
But how does a model that just predicts the next token “call a tool”? Each turn, the harness sends it the instructions, the latest camera images, the robot’s state, and a short description of each tool. The model is trained to reply in a structured format when it wants to use one, and the harness runs it. One turn looks roughly like this:
Harness → model:  "Put the block in the bowl."
                  [front camera image] [wrist camera image]
                  gripper at (0.31, 0.00, 0.40), open
                  tools: move_eef(x, y, z, yaw, gripper), give_up()

Model → harness:  move_eef(x=0.52, y=-0.08, z=0.15, yaw=30, gripper="open")

Harness → model:  "Arrived. Gripper at (0.52, -0.08, 0.15), open."
                  [new camera images]
The model writes a function call, the harness runs the real function, and the result starts the next turn.
Who does what varies by setup. Some harnesses compute joint angles for the model, and others make it give them directly. As Section 7 shows, that choice can swing success a lot, unless an agentic harness smooths it over.

3. Not a new idea

None of this is new by itself. The idea of letting a language model run a robot is about four years old.
The first wave came in 2022. SayCan [9] had the model pick from a fixed menu of skills, such as “pick up the sponge.” Inner Monologue [10] fed back what happened after each step so the model could replan. Code as Policies [11] let the model write the controller itself as code. And VoxPoser [12] had it sketch 3D maps of where the gripper should go, for a motion planner to follow. In every case, the model decided what to do, and something else decided how.
Why didn’t that wave take over? One big reason: the models couldn’t really see. SayCan and Code as Policies ran on text-only models, so all the vision had to be bolted on from outside. Even once models could read images, they stayed weak at the perception a robot needs. On the BLINK benchmark, people scored 95.7% on tasks such as judging which of two points is closer to the camera. GPT-4V scored 51.3%, only about 13 points above random guessing [13].
So the field went the other way. VLAs folded the language model into one big trained policy that maps images and instructions straight to actions, and that became the default, from RT-2 [14] to π0.5 [15].
Then Astra arrived, and the old idea came back, this time with a model capable enough to choose the motion itself across large benchmarks. Next, let’s look at how people are setting it up.

4. How people set it up

“LLM controls robot” covers a lot of very different systems. The clearest way I’ve found to sort them is one question: what does the LLM actually produce each time it’s called? That gives four families.
Figure 2. Four ways to put an LLM in charge of a robot, sorted by what the LLM produces. Code and calls are illustrative.
1. The LLM outputs actions. This is the setup from Figure 1. Every turn, the model looks at fresh images and sends one motion command, and the harness checks it and runs it. The RoboDojo study works this way [2]. GPT-Policy also puts human or robot demo videos into the model’s context, so it can copy what it sees [16]. RoboICL adds a running memory of the model’s own recent moves and what they led to [17]. NVIDIA’s Astranomy study runs Astra inside Codex, a coding agent. Astra can use Python and inspect images along the way, but what it sends the robot is a motion command [18].
2. The LLM outputs programs. Instead of one move at a time, the model writes code, something like “detect the block, plan a grasp, pick, place,” that runs on its own using ready-made perception and control tools. The model reviews the execution feedback and writes the next piece. ASPIRE keeps validated fixes as reusable guidance for future tasks [19]. PyRUA-Lean has the model write short Python programs instead of making one tool call per step [4].
3. The LLM calls learned policies. Here a trained VLA still does the moving, and the LLM manages it. This is the orchestration setup from Section 2 [8], with a twist: the LLM decides when to use the learned policy. Harness VLA wraps a frozen VLA as one tool for the tricky contact-rich moments [20]. Galanti et al. give the LLM a grasp planner and π0.5, and have it pick one for each subgoal, check the result, and retry if needed [21]. A Galbot study flips the roles: π0.5 proposes every move, and Astra accepts, tweaks, or replaces it [22].
4. The LLM improves the policy. Here the LLM isn’t in the control loop at all. In ENPIRE, coding agents run rollouts on real robots, read the logs, edit the training code, and retrain, over and over. The agents even build the reset, success checks, and safety limits themselves, with human feedback. Once those are in place, they run the loop on their own [23].
All four share the loop from Figure 1: act, look, adjust. What changes is what gets adjusted: a single move, a program, a choice of policy, or the policy itself.

5. Why GPT-6 Astra is suddenly good at robot control

So why now? Honestly, nobody outside OpenAI knows for sure. OpenAI hasn’t said what changed in training. What follows is my best read of the evidence. Treat these as educated guesses, not facts. Two caveats up front. The RoboDojo authors couldn’t rule out that some benchmark material leaked into Astra’s training data [2]. And Astra got a written recipe for each task, something the trained policies on the leaderboard never saw [2]. Keep both in mind when you read that #1 ranking.
It sees better. A broad visual AI study tested Astra against five other frontier models on more than 30 vision tasks, several of which matter a lot for robots [24]. Astra scored highest on embodied understanding (82.3, against 77.8 for the next-best model) and 2D spatial reasoning (96.0 vs. 89.1). One particularly large jump was in 2D pose estimation: 75.2, against 52.5. But on tasks that need exact measurements, it still trails specialist vision systems. It’s somewhat behind on depth, and its 3D reconstruction error is more than twice theirs. So it’s great at understanding a scene and still shaky at measuring it. That split shows up again in Section 6.
Computer use may carry over. OpenAI built Astra to operate software through screens and controls, and it scores 72.6% on the computer-use benchmark OSWorld 2.0, against 65.7% for its predecessor [1]. Looking at a screen, deciding where to click, and checking what happened is a lot like looking at a table, deciding where to move, and checking the result. In that sense, the robot is just another interface.
It can learn on the fly, sometimes. Give it a demo video or a memory of its own recent moves, and it sometimes improves without any retraining [16, 17]. On a matched set of 34 RoboDojo tasks, though, adding a demo hurt: 22.9% success with no demo, against 17.9% and 12.9% with two kinds of demos [2]. (That 22.9% comes from the subset, so it isn’t comparable to the 22.5% overall average.)
It corrects itself. When the RoboDojo researchers flipped or hid camera views, offset every move, or reversed the axes, Astra sometimes saw the outcome of a move, revised where it thought things were, and changed its next moves [2]. The authors call this preliminary evidence. My guess is that it comes from how models like Astra are now trained, on long agent tasks where they act, check their work, and fix mistakes as they go.

6. Where it still falls short

There are still plenty of limitations, though.
It’s mostly simulation. Real-robot tests exist, from the RoboDojo study’s own trials [2] to GPT-Policy [16], RoboICL [17], Galanti et al. [21], ENPIRE [23], and RoboCurve’s arm tests [25]. But they cover far fewer tasks and trials than what’s being reported in simulated benchmarks.
It struggles with precision. Astra is weak on precision, fast dynamic motion, two-armed coordination, long-horizon tasks, and memory, exactly where trained policies still win [2]. The precision gap matches what the vision study found [24]: good at understanding a scene, shaky at measuring it.
It isn’t safe yet. When the RoboDojo authors tried real robots, unsafe actions damaged hardware [2], and GPT-Policy’s arms collided with each other [16]. Those were accidents. RoboCurve’s RoboHarm study tested something worse: whether frontier robot policies would follow instructions they should refuse. Controlling a pair of robot arms, Astra carried out the harm in 60 of 100 trials, including stabbing a baby doll in 17 of 20 [26]. It wasn’t alone. Claude Fable 5.1 carried out 34 of 100 and refused 20, all on the doll task, and the VLA MolmoAct2 completed only 6, which the authors put down to capability, not caution [26]. That’s one of the biggest things standing between this and a factory floor, or a home.
It’s expensive and slow. Astra costs $10 per million input tokens and $50 per million output tokens [1]. In Astranomy’s simulated runs, the agentic setup cost about $36 per successful episode, against about $2 for direct control [18]. And every decision takes seconds, which makes real-time control of fast or contact-rich motion hard.
It’s harder to watch, under pressure. OpenAI reports that in tests that explicitly asked Astra to evade monitoring, its reasoning was harder to monitor [1]. That’s a stress test, not everyday behavior, but it matters more once there’s a robot arm attached.

7. What agentic robotics means for your robot data

Here’s the part I think matters most if you’re the one building the dataset. When the model is the policy, the thing you curate changes.
Skills and tools become the dataset. An agent can only do what its tools and skills let it do. Unlike a one-off demo, validated fixes persist as reusable guidance for future tasks [19]. So the question shifts from “how many demos do we have?” to “which tools and skills is the agent missing?”
Demos become something you pick, not pile up. In the RoboDojo study, a demo lowered success, and a text version of it hurt twice as much as one with images and gripper poses [2]. RoboICL combined a demo with a memory of Astra’s own recent moves and gained 20 to 27 progress-score points over zero-shot Astra, depending on the task category [17]. Adding demonstrations alone doesn’t guarantee improvement. How the controller uses that context matters.
Traces are the new episodes. The exchange in Section 2’s code block is a trace: what the agent saw, what it called, and what happened. Log every turn like that, and failures become fuel. Agents that read their own logs can fix their programs [19] or retrain their policies [23], and clustering failed traces shows where plans break, which ties back to our earlier piece on what data your robot actually needs.
The harness is part of the asset too. In Astranomy, under direct control, how actions were represented swung success by 24 points (72.7% with end-effector targets vs. 48.7% with joint positions). With the agentic harness, that gap nearly vanished (86% vs. 84%) [18]. Costs swing too: about $2, $18, and $36 per successful episode for direct control, orchestration, and the full agent loop [18]. Which harness choices matter most deserves its own piece.

Look at your robot episodes in a free Voxel51 workspace

Picking demos by scene and layout, and clustering failed episodes, both start with looking at them. Voxel51 opens MCAP recordings and LeRobot datasets in your browser, plays every camera stream on one timeline, and shows robot state and actions for LeRobot episodes. The free plan holds 3 datasets of up to 10,000 samples each, with 20 GB of storage.
Add 1,000 DROID episodes or 2,378 LeRobot Community episodes to a free Voxel51 workspace, or bring your own MCAP or LeRobot data. Each one uses one of your 3 dataset slots.
Get started free

8. Traces are how you close the gap

So, back to where we started. A model evaluated without any robot-specific fine-tuning would have topped a robot benchmark. That’s real, but it’s early: mostly simulation, not yet safe, still slow, and still expensive.
Traces are how you work on those weak spots. When an episode fails, the trace tells you what went wrong, and each answer points to a different fix. If the agent misjudged where the block was, give it a better perception tool. If it picked the wrong move, give it a better-matched demo. If it picked the right move and the robot botched it, fix the harness. Going after failures one at a time like this might be what unlocks the next generation of robust robot policies.

Frequently asked questions

References

  1. OpenAI, GPT-6 Astra (launch post), September 3, 2026, openai.com/index/gpt-6-astra/
  2. Zhang et al., An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond, arXiv:2609.24170, 2026
  3. RoboDojo leaderboard, robodojo-benchmark.com/leaderboard
  4. Si et al., Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens, arXiv:2610.01939, 2026
  5. RoboMME leaderboard, robomme.github.io/leaderboard.html
  6. Goldberg, K., A Tale of Two Cultures: Can Agentic Coding Close the Gap?, plenary talk, ICRA 2026, Vienna, June 2, 2026, youtube.com/watch?v=7HWIUTcGFQ4
  7. Workshop on Agentic Robotics, CoRL 2026, Austin, TX, November 12, 2026, agentic-robotics-workshop.github.io
  8. Hu et al., What Matters in Orchestrating Robot Policies: A Systematic Study of Hierarchical VLA Agents, arXiv:2606.10267, 2026
  9. Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan), CoRL 2022
  10. Huang et al., Inner Monologue: Embodied Reasoning through Planning with Language Models, CoRL 2022
  11. Liang et al., Code as Policies: Language Model Programs for Embodied Control, ICRA 2023
  12. Huang et al., VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models, CoRL 2023
  13. Fu et al., BLINK: Multimodal Large Language Models Can See but Not Perceive, ECCV 2024, arXiv:2404.12390
  14. Brohan et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, CoRL 2023
  15. Physical Intelligence (Black et al.), π0.5: a Vision-Language-Action Model with Open-World Generalization, arXiv:2504.16054, 2025
  16. Cheng et al., In-Context Robot Learning with VLM Agents (GPT-Policy), arXiv:2609.19138, 2026
  17. Liu et al., RoboICL: Embodied In-Context Learning with GPT-6 Astra, arXiv:2609.34261, 2026
  18. Yang, Khetan & Zhang (NVIDIA SRL), Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy, tech report, Oct 2026, research.nvidia.com/labs/srl/projects/astranomy/
  19. Lu et al., ASPIRE: Agentic /Skills Discovery for Robotics, arXiv:2607.00272, 2026
  20. Zhang et al., Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, arXiv:2607.08448, 2026. Code: RLinf, RPent, github.com/RLinf/RPent
  21. Galanti, Shah & Dao, Addressing the Orchestration Gap in Generalist Robots via Physical Agency, arXiv:2607.21725, 2026
  22. Galbot Team, Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies, arXiv:2609.38537, 2026
  23. Xiao et al., ENPIRE: Agentic Robot Policy Self-Improvement in the Real World, arXiv:2606.19980, 2026
  24. Rasheed et al., Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision, arXiv:2609.35718, 2026
  25. RoboCurve, GPT-6 Astra on Robotic Manipulation, Sep 2026, openai.robocurve.org/gpt-6-astra
  26. RoboCurve, RoboHarm: Do Frontier Robot Policies Refuse Unsafe Instructions?, Sep 18, 2026, robocurve.org/roboharm
Photo of Mohamad Louai Shehab
Mohamad Louai Shehab
Postdoctoral Researcher, University of Michigan Robotics
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab

Try Voxel51 for yourself

Get started for free
Loading related posts...
Talk to an AI expert

© 2026 Voxel51 All Rights Reserved

Terms of ServicePrivacy Policy