Policy evaluation / success rate

Policy evaluation is how you find out whether a learned control policy actually works. Since a policy is the function that maps observations to actions, evaluating it means putting it to the test on a task and measuring what happens, usually by running many independent attempts and tallying the outcomes. The most common headline number that comes out of this is the success rate, the fraction of attempts in which the policy achieves the goal. If a manipulation policy is asked to pick up an object across a hundred trials and succeeds in eighty, its success rate is eighty percent. This single figure is the standard way policies are compared across benchmarks.

The appeal of success rate is its simplicity, but that simplicity is also its weakness. A binary succeeded-or-failed verdict collapses everything about how an attempt went into one bit, so it cannot distinguish a near-miss from a catastrophic failure, or a clean success from a lucky, awkward one. It also says nothing about qualities like smoothness, safety, or efficiency of the motion. For these reasons, careful policy evaluation increasingly supplements success rate with finer-grained metrics that capture execution quality and failure structure, and it pays attention to statistics, since too few trials can make one policy look better than another purely by chance.

Evaluating a policy means running it repeatedly on a task, under conditions chosen to reflect what matters, and recording whether each attempt met a defined success criterion. Aggregating these gives the success rate, and running across varied situations tests how well the policy generalizes. Because outcomes are noisy and each rollout is one sample, a sufficient number of trials is needed to draw reliable conclusions, and comparisons between policies require enough runs to be statistically meaningful. To see beyond the pass-or-fail summary, evaluators may also score execution quality, distinguish types of failure, and check for undesirable behaviors, building a fuller picture than success rate alone provides. Evaluation may happen in simulation, in the real world, or both, and differences between those settings connect it to the sim-to-real gap.

Policy evaluation matters because it is how progress in robot learning is actually judged, and the metrics chosen shape what researchers and engineers optimize for. For anyone building or comparing policies, understanding success rate, and its blind spots, is essential to avoid being misled by a convenient number that hides important differences in behavior. The push toward richer evaluation reflects a broader principle: measuring the right things, with enough statistical care, is what makes claims about capability trustworthy rather than merely impressive.

Success rate is the fraction of evaluation attempts in which a policy achieves the task goal, such as succeeding on eighty of a hundred trials for an eighty percent rate. It is the standard headline metric for comparing control policies.

Because it is binary, it collapses an entire attempt into succeeded or failed, so it cannot distinguish near-misses from total failures or account for smoothness, safety, and efficiency. Richer metrics are needed to capture how and why a policy performs as it does.

Each attempt is a noisy sample, so too few trials can make one policy appear better than another by chance. Running enough attempts is necessary to estimate success rates reliably and to make statistically meaningful comparisons between policies.

Policy, Episode / trajectory / rollout, Long-horizon task, Sim-to-real transfer, Disengagement.