Success detection

Success detection is deciding, for a recorded robot episode, whether the task was actually completed, and storing that decision as an outcome label. The label might come from the operator who recorded the episode, from a scripted check in a simulator, or from a learned model that watches the video and judges the result. Every success rate and every filtered training set is built on top of these labels.

It's the per-episode half of policy evaluation. A success rate aggregates outcomes across many rollouts, and success detection produces each outcome. The two fail differently. A success rate can be statistically shaky because there were too few rollouts, while a single outcome label can simply be wrong, and wrong labels don't average out when they share a cause, such as a detector that can't see the object from one camera angle.

In real-world collection, the operator or a reviewer usually marks the outcome when an episode ends. In simulation benchmarks such as LIBERO, the simulator checks each task's goal conditions directly. Learned detectors are increasingly common at scale. SuccessVQA, from DeepMind in 2023, framed success detection as visual question answering and fine-tuned a vision-language model to answer whether a task was completed, across simulated household tasks, real robot manipulation, and human video. Progress models, a form of learned reward function, score how close an episode is to completion at each step, which also locates where an episode went wrong. Whatever the source, the label is stored with the episode and used to filter training data and compute success rates.

Outcome labels are ground truth for robot learning, and they carry the same risks as any other labels. A failed episode labeled as a success enters the training set as a demonstration of what to do. A success labeled as a failure gets filtered out, along with whatever made it work. In evaluation, a detector that is fooled by the same situation every time biases the success rate in one direction, which no amount of extra rollouts will fix. Spot-checking labels against the final seconds of video, especially for long-horizon tasks, is the simplest guard against label noise in outcomes.

For why a success rate means little without its run count and success criteria, see what a robot success rate leaves out.

Not quite. A reward function scores behavior during learning, often at every step, while success detection produces a final outcome label for a whole episode. Learned progress models sit in between, since a progress score at the end of an episode can be thresholded into a success label.

On familiar tasks and scenes, often yes. Fine-tuned detectors tend to be weaker on new environments, unusual camera angles, and subtle outcomes such as a grasp that holds briefly and then slips, so a sample of their labels should be checked by a person before they are trusted at scale.

For multi-step tasks, partial labels are often more useful than a single flag. Recording which subtasks were completed, or where progress stopped, shows where policies fail and lets teams keep the successful portion of an episode for training.

Policy evaluation / success rate, Reward function, Episode / trajectory / rollout, Failure mining, Label noise, Long-horizon task.