Robot demonstration

A robot demonstration is a recorded example of a task done the way you want a robot to do it, usually by a person controlling the robot through teleoperation. Each demonstration is an episode: camera images and proprioception paired with the action commanded at every step, plus a task description such as "put the cup in the sink." Collections of demonstrations are the training data for imitation learning, which trains a policy to reproduce what it sees.

A demonstration can succeed and still be a poor example. A policy learns everything in it: the pause before the grasp, the regrasp after a slip, an operator's habit of approaching from the left, and a task description that says something slightly different from what happened. Quality also varies by who collected the data. In the robomimic study, the same learning algorithms did noticeably worse on datasets collected by several operators of mixed skill than on data from one proficient operator.

Teams collect demonstrations with leader-follower teleoperation, VR controllers, kinesthetic teaching (moving the robot by hand), or handheld recorders that capture a task without a robot. Each demonstration is stored as an episode in a format such as LeRobotDataset, RLDS, or MCAP. Before training, teams review the set: watching episodes, computing quality scores from the state and action streams, and using embeddings to find near-duplicates and outliers. Research methods now score demonstrations automatically. DemInf, presented at RSS 2025, ranks each demonstration by how much it contributes to the mutual information between states and actions, and training on the top-ranked subset improved imitation policies by 5 to 10% on robomimic, with gains on real ALOHA and Franka setups.

A policy treats every demonstration as a correct example, so the demonstration set is the specification of the behavior you will get. Demonstrations are also the most expensive data in robot learning, since each one takes a person and a robot in real time. Removing hesitant or inconsistent episodes can help a policy more than adding episodes of the same quality, which is why demonstration review belongs before training, not after a failed rollout. Patterns found that way feed failure mining and data triage.

For the quality scores teams compute from state and action streams to decide which episodes to watch first, see triaging demonstrations with quality scores.

Usually no. Keep them out of the imitation training set and label them as failures. Failed attempts are useful for success detection, for evaluation, and for finding the situations where operators struggle. Some methods, such as human-in-the-loop correction, deliberately record failures and the operator's recovery.

A high-quality demonstration completes the task directly, with consistent timing, a clean grasp, and a task description that matches what happened. It also fits the rest of the set: similar camera placement, the same action conventions, and a strategy that other demonstrations of the task share, unless the variation is intended.

Yes. Handheld recorders capture a person doing the task with a gripper-like tool, and first-person human video shows tasks at a much larger scale. Both need extra processing to map human motion onto a robot's body and action space before a policy can learn from them.

Episode / trajectory / rollout, Teleoperation, Leader-follower teleoperation, Imitation learning / behavior cloning, Failure mining, Data triage, Success detection.