The World Doesn’t Pause While Your Robot Thinks: Lessons from CoRL 2026 Day 1
Author
Marketing Manager
Voxel51 helps ML and physical Al teams improve model performance with scalable visual data analysis, curation, and evaluation. Our customers include Bosch, Porsche, Ford, Sony, LG, Walmart, Honda, and the US Gov; while researchers at Google, Microsoft, and NVIDIA use Voxel51 to build state-of-the-art physical Al datasets.
See all articles by Layla Yun
In this article
Key takeawaysA success rate can hide a robot that spends its time waitingDifferent decisions belong on different clocksFaster inference cannot recover information the policy never receivedMemory changes what the present observation meansWhat should change in robot evaluations after CoRL 2026?Questions? We have answers.
In this article
Key takeawaysA success rate can hide a robot that spends its time waitingDifferent decisions belong on different clocksFaster inference cannot recover information the policy never receivedMemory changes what the present observation meansWhat should change in robot evaluations after CoRL 2026?Questions? We have answers.
Try Voxel51 for yourself
What Day 1 of CoRL 2026 revealed about reaction speed, memory, and the architecture of useful robots.
A robot’s next action depends on more than what it sees. It also depends on when it sees it, what happened before, and how much time it has to respond. A moving object may slip beyond reach while a model deliberates. A cooking task may fail because the robot cannot track how long the food has been on the heat.
Across the first day of the 2026 Conference on Robot Learning (CoRL), time repeatedly surfaced as a constraint on robot intelligence: how quickly a policy responds, how it perceives motion, and how it remembers progress over a longer task.
For roboticists who missed CoRL, these talks offer a useful way to think about robot capability beyond whether a task eventually succeeds. How long did the robot spend waiting? What history informed its actions? When something went wrong, could it notice and respond?
Key takeaways
- At CoRL 2026, Hang Zhao showed that once inference time counted toward a 50-second deadline in RoboCasa, a direct frontier-model policy completed 15% of tasks and an agentic policy 20%. Four approaches cleared 60% when thinking time was free.
- Zhao splits robot control into layers that run on different clocks: reasoning over seconds, movement coordination at tens of updates per second, and contact and force control at hundreds of updates per second.
- TEMPO, a CoRL 2026 oral, adds a video-derived motion summary and a compressed history of the robot’s own movement commands to a vision-language-action (VLA) model. On the same pipeline, bottle handover success rose from 38% to 74%.
- MEM pairs short-term video memory with longer-term text summaries, so robot policies can run tasks of up to 15 minutes, such as preparing a grilled cheese sandwich.
- RoboTTT stores interaction history in fast weights that update as the robot acts, so a robot policy can use longer histories without raising per-step inference cost.
A success rate can hide a robot that spends its time waiting
A benchmark can reward a robot for eventually completing a task while overlooking how long it spends waiting for instructions.
Hang Zhao, an Assistant Professor at IIIS, Tsinghua University and Principal Investigator of MARS Lab, explored this in his CoRL 2026 keynote, “Organizing Physical AI Across Timescales.” Working in RoboCasa, a simulated kitchen environment for testing robot behavior. He compared five ways of using a frontier model to control a robot, from generating a program in advance to consulting the model at every step.
When thinking time was free, four approaches completed more than 60% of tasks. Once inference counted toward a 50-second deadline, the direct policy completed just 15% and the agentic policy 20%. Both called the model at every step; the agentic policy also allowed extra reasoning and tool use.
Each call consumed time the robot needed to finish. These approaches looked capable when the clock excluded deliberation, then struggled when thinking and acting shared the same deadline.
Zhao’s response was to divide responsibility: let slower models reason about the task while faster controllers handle movement and contact. That raises the question: which decisions can afford to wait, and which need an immediate response?
Different decisions belong on different clocks
Clearing a table involves decisions with very different deadlines. A robot can take a few seconds to choose which object to pick up next. Once its fingers make contact with a wine glass, it must track their relative motion much more frequently and needs faster feedback to regulate force and maintain its grip. A useful architecture has to support both, with information flowing between them.
Zhao organized these responsibilities into three layers, with each layer running at the pace of events it must respond to.
- S2 Task planning: seconds to minutes, responding to instructions, task state, and broader scene changes.
- S1 Movement coordination: tens of updates per second, tracking motion between the hand and an object.
- S0 Contact and force control: hundreds of updates per second, responding to physical interaction.
These deadlines give each layer a clear responsibility. The planner can deliberate over the next task while faster controllers keep the robot responsive to motion and contact.
| Level | Frequency | Reacts to | Typical models |
|---|---|---|---|
| System 2 | Seconds - minutes | Task state, language, scene | Astra, Gemini Robotics-ER, coding agents |
| System 1 | Tens of hertz | Hand-object relative motion | π0.5, G0.5, World Action Models |
| System 0 | Hundreds of hertz | Contact and force | SONIC, Beyond |
Those layers also need to communicate. For instance, if the robot’s hands slip while lifting a box, the controller must report the failed lift so the planner can choose a recovery action.
Faster inference cannot recover information the policy never received
Speed helps only if the policy has enough information to choose an appropriate action. Consider taking a bottle from someone walking past a robot. A faster snapshot gives the robot a more recent view of the bottle’s position, but still leaves it without direct evidence of motion.
- Motion ambiguity leaves the policy unsure where an object is going.
- State aliasing occurs when different task stages look similar but require different actions.
A gripper near a plate might need to pick it up or withdraw after placing it down. Without history, the policy can oscillate between those actions.
TEMPO: Learning Temporal Context for Dynamic Robot Manipulation, a CoRL 2026 oral from researchers at the University of California, Irvine, tackles this by supplying two distinct signals. One is a video-derived summary of scene motion, and the other is a compressed history of the robot’s own movement commands. In bottle handover tests, the underlying policy succeeded in 38% of trials. Adding only movement history raised success to 52%. Adding only visual motion information raised it to 56%. Combining both reached 74%. Each signal supplied information the other could not fully replace.
It is useful to distinguish a late decision from an uninformed one. Both can produce a missed grasp, but they call for different interventions.
Memory changes what the present observation means
Some decisions depend on events much further back than the last few frames. Cooking requires tracking how long food has been on the heat. Cleaning requires remembering which areas are finished. Recovering from a mistake requires knowing what has already been tried.
At CoRL, Marcel Torne, a robotics researcher at Stanford University, illustrated these demands while presenting his team’s work on MEM: Multi-Scale Embodied Memory for Vision Language Action Models. A policy without memory left a grilled cheese sandwich on the heat until he intervened to keep it from burning. In another example, a robot repeatedly cleaned the same spot on a plate because it could not track what it had already done.
MEM combines short-term video memory with longer-term textual summaries. Recent frames preserve details needed for local interactions, while summaries retain earlier events without carrying the full video history. Torne reported that the system could perform tasks lasting up to 15 minutes, including preparing a grilled cheese sandwich and cleaning up the kitchen.
He also showed how memory supported recovery: a policy without memory repeated an unsuccessful action, while a policy with memory used the previous attempt to try a different strategy. The team trained this behavior on human corrections, in which a person steps in after a failure and demonstrates a better approach. Remembering task history helped the robot change what it did next.
Yunfan Jiang’s presentation of RoboTTT: Context Scaling for Robot Policies explored another approach: compressing interaction history into “fast weights” that update as the robot acts. This lets the policy use longer histories without increasing the inference cost of each new step.
The team gave mistakes and corrections different roles in training. Both updated the fast weights, which carried the history of what the robot had tried and what happened next. But only human corrections supplied the imitation targets used to train the slow weights, the model’s persistent parameters. The robot’s mistakes informed its memory without becoming actions it was trained to copy.
This trained the policy to choose a correction in the context of a failed attempt. During deployment, the slow weights stayed fixed while the fast weights continued updating, allowing the robot to use its experience from previous attempts to adjust its next action.
In one assembly example, the robot missed a screw, raised its arm, and repositioned the drill. After missing again, it adjusted once more and succeeded. The recovery showed why history matters: a failed attempt could guide the next action instead of leaving the robot to repeat it or move on prematurely.
What should change in robot evaluations after CoRL 2026?
These talks point toward concrete changes in how we evaluate robots. Count inference time toward task deadlines. Test moving targets at different speeds and directions. Retain task stages that look similar but require different actions. Introduce failed grasps and unsuccessful attempts, then examine whether the robot recognizes the failure and changes its approach.
Alongside completion rates, measure time spent waiting, repeated attempts, and successful recoveries. For layered systems, check whether execution failures reach the planner before it chooses the next step. These measures help distinguish a robot that finishes under favorable conditions from one that can keep a task moving when conditions change.
The world keeps moving while a robot thinks. Useful robots need to keep pace—and carry enough of the past forward to choose what comes next.
Questions? We have answers.
A robot policy that calls a large model at every step spends time the task needs. In the RoboCasa results Hang Zhao showed at CoRL 2026, a direct policy completed 15% of tasks and an agentic policy 20% once inference counted toward a 50-second deadline.
Zhao's approach runs reasoning over seconds, movement coordination at tens of updates per second, and contact and force control at hundreds of updates per second. The layers also have to communicate, so a failed lift reaches the planner before it picks the next step.
Many tasks depend on events further back than the last few frames, such as how long food has been on the heat. In Marcel Torne's CoRL 2026 talk on MEM, a policy without memory left a grilled cheese sandwich on the heat until he intervened.
They should count inference time toward task deadlines and track time spent waiting, repeated attempts, and recoveries alongside completion rates. They should also test moving targets, similar-looking task stages, and deliberately failed attempts.
Marketing Manager
Voxel51 helps ML and physical Al teams improve model performance with scalable visual data analysis, curation, and evaluation. Our customers include Bosch, Porsche, Ford, Sony, LG, Walmart, Honda, and the US Gov; while researchers at Google, Microsoft, and NVIDIA use Voxel51 to build state-of-the-art physical Al datasets.
See all articles by Layla Yun
Try Voxel51 for yourself
Loading related posts...
© 2026 Voxel51 All Rights Reserved
