How to Collect Teleoperation Data for Robot Learning

When I was learning how to write in cursive for the first time, my mom took great pleasure in guiding my hands to trace the word "apple." My earliest education often relied on me observing someone else carry out actions first before trying to replicate it myself. More generally, as humans learn the kinematics and dynamics of our bodies, we can generalize these capabilities to specialized contexts. Modern robotics builds on this experience as the basis for teleoperation.

Grid of eight teleoperated robots, including the Robonaut 2 and TALOS humanoids, a Unitree Go2 quadruped, a Stretch mobile manipulator, and a Nano Hummingbird drone. — Figure 1. Teleoperation spans many kinds of robots, including drones, humanoids, quadrupeds, and manipulators. Source: IEEE Spectrum ROBOTS guide (https://robotsguide.com/robots?feature=teleoperated&sort=name)

Key takeaways

Teleoperation data teaches robots through demonstrations, so every bias, delay, and mislabeled episode in the dataset becomes behavior a policy can learn.
For scene generalization, the DROID dataset showed that a consistent collection protocol across varied scenes mattered more than raw dataset size.
Teleoperation latency changes how operators move, so recording raw operator commands alongside executed actions lets you find demonstrations that latency distorted.
Labeling failed teleoperation episodes, stalls, and resets, rather than deleting them, keeps recovery data and lets you trim bad segments instead of discarding whole episodes.
Auditing a teleoperation dataset for operator, task, and scene balance before training catches skews that one operator or scene would otherwise teach a policy.

How teleoperation teaches robots

Teleoperation, the remote control of robots, relies on consistent operator feedback to actuate any humanoid, manipulator, or car toward a desired goal state or action state. The intermediate states and modalities become data that a robot can learn from to complete the same task autonomously, even without a step-by-step algorithm. As you show the robot how to do the task, the robot can do what it does best: recognize the underlying mathematical patterns in decision making from the demonstrations.

Plan your setup for good data

Diagram of a teleoperation setup in which a human operator uses a haptic device and computer at the leader site, connected over the internet or a cable line to a follower robot and computer at the follower site. — Figure 2. A typical teleoperation setup has a leader site that the operator controls, communication channels that carry commands to the robot, and a follower site where the robot carries out the task. Source: Yang, Luo, and Wang, Human-in-the-Loop Learning and Control for Robot Teleoperation (https://doi.org/10.1016/B978-0-32-395143-2.00005-X)

A typical teleoperation experiment uses a communication medium between the leader and follower sites (Figure 2). The human operator uses a joystick, haptic device, exoskeleton, or other interface to send signals to a follower robot. Based on the hardware integration available on the robot, you can then record sensor and other peripheral measurements that characterize the robot's states in its motion. Human input and sensor integration naturally introduce biases and noise, so choices made before your first trial determine your dataset's limitations.

Balance consistency against variety

Similar to scouring blogs and cookbooks for the recipe to the perfect soufflé, learning-based methods consume vast training datasets, often scaling collaboration across multiple universities and research bodies. Variation allows models to perform closer to the intended action in environments with distractions or unseen objects. This is known as generalization.

Consider two massive manipulator datasets: the Open X-Embodiment Dataset (OXE) and the Distributed Robot Interaction Dataset (DROID). OXE consists of over 1M demonstrations with 22 different robots across 21 institutions. DROID is less than a tenth of that size, with over 76k trajectories, but it spans 564 scenes to OXE’s roughly 300, all recorded on a standardized hardware setup by 50 data collectors in North America, Europe, and Asia.

Despite its smaller size, co-training with DROID outperformed other methods, including co-training with OXE, by 22% on in-distribution tasks and by 17% on out-of-distribution tasks. DROID also isolated the effect of scene variety. When it compared 7,362 trajectories from its 20 most-collected scenes against the same number sampled across all scenes, the more diverse sample performed better on out-of-distribution tasks. For scene generalization, a consistent protocol with varied scenes mattered more than raw scale. Because OXE spans 22 robots, it may better suit embodiment generalization. Whatever you optimize, pair a varied dataset with a consistent collection protocol.

Record enough context to reuse every trial

Large, varied datasets need contextual metadata for each trial, so that you can spot biases and reconcile differences between trials. Common examples include timestamps, operator IDs, trial durations, and workspace objects (position, size, shape, color, and so on).

Record the states your robot encounters so you can analyze or replay them sequentially. These may include sensor inputs and outputs, joint angles, intermediate camera frames, trajectory plans, and control inputs and outputs. Record the input and output of every intermediate hardware and software component, from sensor and camera feedback to function calls. Every later step in this guide, from filtering to balance audits, depends on what you record here.

Reduce noise and delay in your data

All communication channels naturally have some noise and delay as an inevitable property of the hardware (wiring, coupling, circuitry, and so on) that enables data transmission. When working with teleoperation interfaces, certain considerations can help you account for these properties.

Recalibrate to prevent drift

Calibrate all hardware before you start any experimentation. While daily calibration is a good start, analyze your data periodically and recalibrate after enough trials. Temperature changes, overheating, and other factors cause drift that compounds into measurement errors, which a model may learn as real.

Measure latency across your pipeline

Break your pipeline into hardware and software components and measure each latency from human action to robot reaction. How much latency you can tolerate depends on the application, from under 50 ms for remotely driving a car to as much as 250 ms for telestenting. For your dataset, what matters is how latency changes operator behavior. Under high latency, operators tend to fall into a "move-and-wait" pattern: give an instruction, observe feedback, and then give the next instruction. This leads to long trials and trajectories that are not smooth. A policy trained on those trials learns to hesitate. Record raw operator commands alongside executed actions to identify where latency distorted a demonstration.

Smooth out jittery actions

Teleoperation interfaces also add noise through non-uniform acceleration, sporadic starts and stops, and oscillations. Spot-check random trial windows, and fix noisy batches before collecting more data. Some of this jitter comes from retargeting, the mapping of an operator's actions onto the robot, and methods such as Sampling-Based Retargeting smooth these trajectories at the source. Whatever smoothing you apply, log both the raw and smoothed signals, because the smoothed version is what your policy will imitate.

Clean your teleoperation data before training

Collecting data is only half the work. Once trials are recorded, ensure the data is aligned, labeled, and filtered before training. Problems caught here are far cheaper to fix than those discovered after training.

Align your sensor streams

With multimodal data, temporal synchronization lets sensors and cameras with varying latencies capture the system together. Even relative delays of a few milliseconds can cause a model to train on misaligned triggers, such as a gripper closing before a camera shows it touching the object.

While some devices such as RealSense cameras already come with hardware synchronization, you may need to implement your own software synchronization for others. For instance, the Universal Manipulation Interface (UMI) requires GoPros connected to the manipulators to record a QR code with a global clock before runs to calculate latencies. During a dynamic tossing task, when the relative latencies were not adjusted for, the success rate of an accurate toss dropped from 87.5% to 57.5%. DROID also embeds this consistency in its pipeline by ensuring that all cameras operate at the same 15 Hz frequency. For contact-rich events, force-feedback outputs can also be analyzed with visual cues to ensure multimodal feedback aligns.

Label successes, failures, and stalls

Define success for each task quantitatively. For instance, a drawer-opening task could be considered successful if the drawer opened at least 1 inch from its starting position. Concrete definitions let multiple operators annotate consistently with less human bias. Use the same metrics to label failed trials rather than deleting them. DROID kept roughly 16k unsuccessful trajectories in its release alongside its 76k successful ones. These trials matter because a policy trained only on clean demonstrations never sees how to recover after drifting off course. Near-misses and operator corrections are exactly the recovery data that fills this gap.

Beyond successes and failures, label stalls and intermediate states such as "reaching," "grasping," and "releasing." A stall could be a latency issue or an operator hesitating, and if those near-zero actions stay unlabeled, a policy may learn to get stuck in the same spot during execution. OpenVLA avoids this by removing all-zero actions from its training data. When an operator takes over to correct a robot, Sirius labels the robot's actions before the intervention as lower quality and the human correction as data to learn from, so its model can weigh the correction more heavily. Keep resets out of the episode entirely, either by labeling them or by recording them in a separate phase. LeRobot's recording tool does this with a dedicated reset window between episodes. These labels let you trim the segment where a trial went wrong instead of discarding the whole episode.

For larger datasets, automate annotation. SPARC uses gripper-state phases to segment demonstrations automatically. Critic in the Loop combines sudden changes in end-effector trajectories with a vision-language model (VLM) to label segments with semantic descriptions. Still, spot-check automated flags because a trajectory can look complete even when the object slips at the last second. In one simulated study, 47.6% of episodes flagged as successful on a transfer task were actually failures. Multiple reviewers can cross-check the same episodes and use a majority vote to settle ambiguous success labels.

Filter out behavior you don't want learned

Filter behavior that does not reflect the intended task before training. Once your labels are in place, many filters become simple rules. The authors of SOMA, a spatial memory framework for out-of-vision manipulation, rejected trajectories with 300 ms or more of input lag, obstructed camera views, or dropped frames that broke alignment between video and joint data. Design rules around what your demonstrations should teach, and flag episodes far outside the task's average duration.

Even after you remove broken episodes, not every remaining demonstration is equally useful. Picking up a mug by the handle in some trials and by the rim in others may confuse the model. Belkhale et al. found that demonstrations handling similar situations with inconsistent actions tend to hurt policy performance. A strict protocol that defines goals, combined with spot-checks for consistency, helps here. To find noisy demonstrations without reviewing every trial by hand, you can also rank them with a quality metric. Sojib et al. scored demonstrations by the spectral power of the end-effector trajectory, where power concentrated at higher frequencies indicates jittery movement. Filtering this way took seconds and raised a fine-tuned policy's success rate from 15% to 38%. Similarly, Demo-SCORE used a policy's failed rollouts to identify which demonstrations to omit, raising success rates by 15% to 40% on four real-world tasks.

Audit operator, task, and scene balance

A pipeline built for variety may still produce an unbalanced dataset. Count how many episodes each operator, task, and scene contributes. One operator or scene may dominate the data, causing a model to learn their quirks as part of the task.

Operator skill level is often overlooked. In robomimic, policies trained on 300 demonstrations from six operators of mixed experience performed worse on harder tasks than policies trained on 200 demonstrations from a single experienced operator. The least experienced operators' demonstrations were also about twice as long, so comparing average trial duration per operator is a quick way to spot skill gaps.

The fix is not always to throw out data. You can collect more episodes for underrepresented scenes and tasks, give struggling operators clearer guidelines and more practice, or reweight your data during training. On Open X-Embodiment, learned weights for each of its component datasets outperformed uniform weighting by 38%. Reweighting subsets of your own data, such as by scene or operator, can improve performance without collecting anything new.

Afterword: catch mistakes early

When my mom guided my hand through "apple," the quality of that demonstration shaped what I learned. Robots are no different. The guide follows one idea: the earlier you catch a mistake, the cheaper it is to fix. A drifted calibration or unsynchronized camera takes minutes to fix during collection but can later cut success and force retraining. A mislabeled failure is easy to relabel during review, but once it is in your training set, it may hide among thousands of episodes that look successful. So build checks into every stage, from spot-checking trials as you collect to auditing your dataset before a model ever sees it. The robot learns exactly what you showed it, so show it carefully.

If you want to run these checks on your own teleoperation data, Voxel51 can load LeRobot datasets natively, show multi-camera episodes side by side as grouped datasets, and filter episodes by the metadata you recorded. You can start for free at app.voxel51.com with no credit card required. The free tier includes 3 datasets of up to 10,000 samples each and 30 hours of compute per month.

Frequently asked questions

Teleoperation data is the record of a human operator remotely controlling a robot through a task, including the operator's commands, the robot's joint states, and synchronized camera and sensor streams. A robot learns from these demonstrations to complete the same task autonomously, without a step-by-step algorithm.

Acceptable teleoperation latency depends on the application, from under 50 ms for remotely driving a car to as much as 250 ms for telestenting. For dataset quality, high latency matters because it pushes operators into a move-and-wait pattern that produces long trials and trajectories that are not smooth.

No, label failed teleoperation episodes instead of deleting them. DROID kept roughly 16k unsuccessful trajectories in its release alongside its 76k successful ones, and near-misses and operator corrections give a policy the recovery data that clean demonstrations lack.

You can filter low-quality teleoperation demonstrations with rules built on your labels, such as rejecting episodes with 300 ms or more of input lag, obstructed camera views, dropped frames, or durations far outside the task average. Quality metrics, such as the spectral power of the end-effector trajectory, can also rank demonstrations by jitter without a manual review of every trial.

No, a bigger teleoperation dataset is not always better for scene generalization. DROID is less than a tenth the size of Open X-Embodiment, yet co-training with DROID outperformed co-training with Open X-Embodiment on out-of-distribution tasks because it pairs varied scenes with a consistent collection protocol.

Sources

Robots Team. "Explore Robots." ROBOTS: Your Guide to the World of Robotics. IEEE Spectrum. Accessed September 30, 2026. https://robotsguide.com/robots?feature=teleoperated&sort=name.
Yang, Chenguang, Jing Luo, and Ning Wang. "Introduction." In Human-in-the-Loop Learning and Control for Robot Teleoperation, 1–15. Academic Press, 2023. https://doi.org/10.1016/B978-0-32-395143-2.00005-X.
Heidaripour, Sara. "Interfaces for Learning from Human Demonstrations." RAS University. IEEE Robotics and Automation Society. Accessed September 30, 2026. https://www.ieee-ras.org/ras-university/?ras_page=docs/chap13_hri/interfaces-for-LfD.html.
O'Neill, Abby, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, et al. "Open X-Embodiment: Robotic Learning Datasets and RT-X Models." In 2024 IEEE International Conference on Robotics and Automation (ICRA), 6892–6903. IEEE, 2024. https://doi.org/10.48550/arXiv.2310.08864.
Khazatsky, Alexander, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, et al. "DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset." In Robotics: Science and Systems (RSS), 2024. https://doi.org/10.48550/arXiv.2403.12945.
T-Systems. "Teleoperated Driving: Safely Controlling Cars Remotely." T-Systems. Accessed September 30, 2026. https://www.t-systems.com/dk/en/industries/automotive/solutions/teleoperated-driving.
Madder, Ryan D., Stacie VanOosterhout, Abbey Mulder, Jared Bush, Samuel Martin, Adam J. Rash, Jose Mariano Tan II, et al. "Network Latency and Long-Distance Robotic Telestenting: Exploring the Potential Impact of Network Delays on Telestenting Performance." Catheterization and Cardiovascular Interventions 95, no. 5 (2020): 914–919. https://doi.org/10.1002/ccd.28425.
Ferrell, William R. "Remote Manipulation with Transmission Delay." IEEE Transactions on Human Factors in Electronics HFE-6, no. 1 (1965): 24–32. https://doi.org/10.1109/THFE.1965.6591253.
Malate, Robert Jomar, Erik Bauer, Norica Bacuieti, Stefanos Charalambous, Elvis Nava, Robert K. Katzschmann, and Benedek Forrai. "Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting." arXiv preprint arXiv:2607.07491 (2026). https://doi.org/10.48550/arXiv.2607.07491.
Wang, Xinyi, Donghan Li, Zi'Ang Chen, Chong Yu, Chen Xin, Peng Ye, Yingkai Sun, and Tao Chen. "RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation." arXiv preprint arXiv:2606.31836 (2026). https://doi.org/10.48550/arXiv.2606.31836.
Chi, Cheng, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. "Universal Manipulation Interface: In-the-Wild Robot Teaching without In-the-Wild Robots." In Robotics: Science and Systems (RSS), 2024. https://doi.org/10.48550/arXiv.2402.10329.
Xu, Yiwen, Liangtao Huang, Tiesong Zhao, Ying Fang, and Liqun Lin. "A Timestamp-Independent Haptic–Visual Synchronization Method for Haptic-Based Interaction System." Sensors 22, no. 15 (2022): 5502. https://doi.org/10.3390/s22155502.
Nair, Suraj, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. "Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation." In Conference on Robot Learning (CoRL), 2021. https://doi.org/10.48550/arXiv.2109.01115.
Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning." In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 627–635. PMLR, 2011. https://doi.org/10.48550/arXiv.1011.0686.
Kim, Moo Jin, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, et al. "OpenVLA: An Open-Source Vision-Language-Action Model." In Proceedings of the 8th Conference on Robot Learning (CoRL), 2679–2713. PMLR, 2025. https://doi.org/10.48550/arXiv.2406.09246.
Liu, Huihan, Soroush Nasiriany, Lance Zhang, Zhiyao Bao, and Yuke Zhu. "Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning during Deployment." In Robotics: Science and Systems (RSS), 2023. https://doi.org/10.48550/arXiv.2211.08416.
Hugging Face. "Imitation Learning on Real-World Robots." LeRobot Documentation. Accessed September 30, 2026. https://huggingface.co/docs/lerobot/il_robots.
Blank, Nils, Paul Mattes, Maximilian Xiling Li, Jakub Suliga, Thomas Roth, Moritz Reuss, Pankhuri Vanjani, and Rudolf Lioutikov. "SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale." arXiv preprint arXiv:2606.13497 (2026). https://doi.org/10.48550/arXiv.2606.13497.
Yi, Pengfei, Yingjie Ma, Wenjiang Xu, Yanan Hao, Shuai Gan, Wanting Li, and Shanlin Zhong. "Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation." arXiv preprint arXiv:2603.05185 (2026). https://doi.org/10.48550/arXiv.2603.05185.
Bedi, Aarav. "How Visible Are Silent Manipulation Failures? An Observability Study of False-Success Detection in Simulated Robot Episodes." arXiv preprint arXiv:2606.03134 (2026). https://doi.org/10.48550/arXiv.2606.03134.
Du, Yuqing, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. "Vision-Language Models as Success Detectors." In Proceedings of the 2nd Conference on Lifelong Learning Agents (CoLLAs), 120–136. PMLR, 2023. https://doi.org/10.48550/arXiv.2303.07280.
Li, Pengteng, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, and Hui Xiong. "Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action." arXiv preprint arXiv:2605.22283 (2026). https://doi.org/10.48550/arXiv.2605.22283.
Belkhale, Suneel, Yuchen Cui, and Dorsa Sadigh. "Data Quality in Imitation Learning." In Advances in Neural Information Processing Systems (NeurIPS), 2023. https://doi.org/10.48550/arXiv.2306.02437.
Sojib, Noushad, Sajay Arthanat, and Momotaz Begum. "An Efficient Metric for Data Quality Measurement in Imitation Learning." arXiv preprint arXiv:2605.01544 (2026). https://doi.org/10.48550/arXiv.2605.01544.
Chen, Annie S., Alec M. Lessing, Yuejiang Liu, and Chelsea Finn. "Curating Demonstrations Using Online Experience." In Robotics: Science and Systems (RSS), 2025. https://doi.org/10.48550/arXiv.2503.03707.
Mandlekar, Ajay, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. "What Matters in Learning from Offline Human Demonstrations for Robot Manipulation." In Conference on Robot Learning (CoRL), 2021. https://doi.org/10.48550/arXiv.2108.03298.
Hejna, Joey, Chethan Bhateja, Yichen Jiang, Karl Pertsch, and Dorsa Sadigh. "Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learning." arXiv preprint arXiv:2408.14037 (2024). https://doi.org/10.48550/arXiv.2408.14037.

How to Collect Teleoperation Data for Robot Learning

Try Voxel51 for yourself

Key takeaways

How teleoperation teaches robots