Author
Technical Writer
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab
In this article
Try Voxel51 for yourself
Key takeaways
- The sim-to-real gap almost always traces back to one of three buckets: dynamics, perception, or task distribution.
- Domain randomization hedges against the sim-to-real gap by widening the training distribution, but randomizing everything wastes capacity and can produce a worse, overly conservative policy.
- Before blaming the sim-to-real gap, measure the simulation success rate with a confidence interval, since nine successes in ten rollouts is consistent with anything from about 56% to 99.7%.
- Five hardware failure patterns, from early-step action collapse to workspace degradation, each point to a specific subsystem and a cheap check that localizes the sim-to-real gap.
- Offline checks on sim and real frames, such as raw image statistics, embedding centroid distance, and linear probes, can find observation-side sim-to-real bugs without moving the robot.
The problem
Locomotion made this look solved
At CoRL 2021, an ETH Zurich and NVIDIA team put an ANYmal quadruped on stage beside a single workstation and launched training live during the talk, running thousands of simulated robots over varied terrain in Isaac Gym. Every 15 seconds, roughly 8 hours of simulated time, the latest policy was pushed to the physical robot. Within minutes, the robot was walking. Since then, quadruped locomotion has largely been considered solved.
Yet anyone who has worked on tabletop manipulation knows the more typical ritual: days of training in simulation, a huddle around the emergency stop (E-stop), and a first eval run in which a multi-thousand-dollar arm sweeps down with precision, closes cleanly on the mug, and lifts it just far enough to watch it slide out of its fingers.
But why is that the case?
Fundamentally, it's about how forgiving these tasks are of wrong assumptions. Think about how often you have misjudged the last step of a staircase without ending up on the floor. Now think about the last time you picked up a glass without realizing it was wet, and how quickly it left your hand. In both cases you could see perfectly well. The difference is not sensing. It is what a wrong contact model costs you. A quadruped has twelve actuated joints, several feet sharing the load, a gait in which every step is a new chance to place a foot, and a control loop fast enough to correct a bad one on the next stride. Contact error here is absorbed by the redundancy in the quadruped design. Manipulation, on the other hand, concentrates the whole task into one or two contact patches with no redundancy behind them. The friction coefficient at the fingertip is not a disturbance on the task. It is the task.
The same pattern shows up beyond robotics: methods succeed first where the domain forgives their assumptions. Across 155 deterministic benchmark Markov decision processes (MDPs), deep reinforcement learning (RL) tends to succeed where a task needs little lookahead beyond the random policy's Q-values. The environment's structure did the work, and the field credited the method. Sim-to-real cleared locomotion because locomotion is cheap to be wrong about.
The sim-to-real gap and domain randomization
When a policy trained in simulation fails on hardware, we blame the sim-to-real gap. The cause is almost always in one of three buckets. Dynamics: contact and friction models, link masses and inertias, joint damping, actuator lag and torque limits, transmission compliance, controller gains, and control rate. Perception: lighting and shadows, material textures and reflectance, background and table appearance, camera pose and intrinsics, exposure and white balance, depth noise and dropout on reflective surfaces. Task distribution: object pose and orientation ranges, object instances and their geometry, mass and size variation, clutter and distractors, workspace coverage of initial states, goal placements, and episode length or horizon.
Domain randomization is the standard response to the sim-to-real gap, and it works often enough to be the default. Randomize textures and lighting widely enough, and a detector trained without a single real image transfers to real scenes. Randomize masses and friction, and a pushing policy transfers to a physical arm.
What domain randomization is really doing is widening the training distribution until the real world falls inside it. It works because policies, whether trained by imitation or by RL, are reliable mostly on the states their training actually covered and degrade as deployment drifts away from that support.
Widening the distribution hedges against all three buckets at once. That is also its blind spot. Randomize all three together and capacity goes to variation that does not matter, and past a point you get a worse policy, one that has learned the conservative behavior fitting every unnecessary variation you added rather than the one that does the task well.
So after a zero-shot failure, the way forward is not more randomization. It is finding out, as cheaply as possible, which bucket you are in.
Before you diagnose
Get the setup right
Everything below assumes the unglamorous part is done. Your Unified Robot Description Format (URDF) file should carry measured masses, inertias, joint limits, and damping, not rounded spec-sheet values. Objects should match on geometry and texture, cameras on intrinsics and extrinsics, and those should be rechecked, since a bumped camera mount quietly undoes a good calibration. The action space, control frequency, and low-level controller should be identical in both places, since a policy trained against a 50 Hz position interface and deployed on a 200 Hz torque interface is not the same policy.
This matters because the items above are all specification problems, and a simulator reproduces them faithfully once you specify them faithfully. However, not everything in the dynamics bucket is like that. Friction and compliance are not parameters you look up and enter. They emerge from surface roughness and material deformation that no simulator resolves directly, which is why contact is the one place where careful setup still leaves you short. Collins et al. found that five physics engines modeled kinematics and control accurately against motion capture, with their deficits concentrated in dynamic robot-object interaction.
This setup work does not close the gap. It stops you from misattributing an avoidable error to an unavoidable one.
Establish the simulation number first
The cheapest check is the one people skip: the policy's simulation success rate, with a confidence interval. Nine of ten rollouts is not a 90% policy. The exact 95% interval runs from about 56% to 99.7%. Nine hundred of a thousand gives roughly 88% to 92%, and only that supports "it works in sim but not on hardware." Otherwise, a 40% hardware rate might be a reality gap, or a policy that was never as good as you thought, and you are about to spend a month debugging physics to explain a training problem. Run enough rollouts, in sim and on hardware, that both intervals are narrower than the drop you are trying to explain.
Order your evaluations by how much the task forgives
Karpathy's advice for debugging neural networks is to start simple and add complexity one piece at a time. The hardware version is to rank tasks by how much positional and force error they tolerate. Centimeter clearance and millimeter insertion are not two points on one difficulty curve. They are different diagnostic instruments. Start with the forgiving one. If it also fails, the problem is upstream of precision contact. If it succeeds and the tight one fails, you have localized the gap to precision-critical dynamics.
Diagnosing the sim-to-real gap
A taxonomy of hardware failures
With a trustworthy simulation number and a forgiving-to-tight task ordering, a hardware failure stops being an anecdote and becomes evidence: how and where the policy fails narrows the cause far more than whether it fails. The five patterns below cover a lot of what we see. Two trace back to the observation side, two to dynamics, and one sits between calibration and task coverage. For each, we give the observed failure, its likely cause, and a check.
Pattern 1: Early-step action collapse
Observed failure: The policy emits degenerate or near-constant actions from the first step, before any meaningful motion.
Likely cause: An observation plumbing mismatch between simulation and hardware: wrong normalization statistics, swapped channel order (RGB vs. BGR), a different resize filter, or mismatched proprioceptive units.
Check: Push one recorded real observation and one matched simulated observation through the frozen policy and compare the actions. Then compare the raw tensors using the raw statistics checks in the offline checks section below. A sharp difference means a pipeline bug, fixable without retraining or touching physics. Plumbing defects corrupt every observation identically, so the collapse looks the same in every scene.
Pattern 2: Trajectory tracking lag and offset
Observed failure: The executed trajectory has the right shape, but the end-effector is consistently offset or lags the commands.
Likely cause: Low-level tracking problems, such as controller gains, actuator latency, unmodeled motor dynamics, or a rate mismatch between the policy loop and the motor drivers.
Check: Replay joint command sequences spanning the workspace in sim and on hardware with no policy in the loop, and compare joint tracking error against task tolerance. With perception removed, any discrepancy is a control or dynamics defect. Actuator modeling was a large part of the ANYmal result we opened with: the pipeline learns a network model of the series elastic actuators, which resist analytic modeling.
Pattern 3: Contact transition instability
Observed failure: The policy reaches the object smoothly and fails at contact by slipping, penetrating, or knocking it away.
Likely cause: Inaccurate contact physics modeling in simulation, specifically miscalibrated coefficients of friction, surface compliance, or restitution parameters.
Check: Use the forgiving-to-tight task ordering from earlier. If the forgiving variant succeeds, the tight one fails, and Pattern 2's tracking error is within tolerance, contact modeling is what remains. For a direct test, replay a scripted open-loop grasp or push in sim and on hardware and compare the object's trajectory. Divergence after first contact points to the contact model. For contact-rich tasks, consider adding a simulated tactile or force-torque sensor.
Pattern 4: Visual sensitivity
Observed failure: Success varies across geometrically equivalent scenes. The pose and the reach are the same, but changing the lighting, background, table covering, or object instance flips the outcome.
Likely cause: Perception domain shift. Real images differ from rendered ones in ways training never covered, and whatever reads them, whether a pose estimator or the policy's own visual encoder, has latched onto features that exist only in simulation.
Check: If your policy consumes object poses, replace the perception pipeline on hardware with ground-truth poses from motion capture or fiducials such as AprilTags. A sharp jump in success localizes the gap to the perception stack. End-to-end image policies have nothing to swap, so start offline with the embedding comparison in the offline checks section below, then confirm on hardware by holding the scene geometry fixed and changing one visual factor at a time: lighting, then table covering, then object instance. If a single factor flips the outcome, that is where your visual randomization budget should go. Either way, the diagnostic signal is variance across scenes. Identical failure everywhere points to Pattern 1.
Pattern 5: Workspace inconsistency and spatial degradation
Observed failure: The policy is reliable in one region of the workspace and degrades toward the edges or other quadrants.
Likely cause: Either an incorrect extrinsic transform between camera and robot base, or training data that never covered those regions. Both correlate with position, so the failure pattern alone cannot separate them.
Check: First, reproject known 3D points into both sim and real cameras across a workspace grid. Calibration error is structured, typically growing away from the calibration region, so the error surface is the signature. Then plot your training initial-state distribution on the same grid. If reprojection error is flat and failures fall where coverage is thin, you have a distribution shift, which belongs in your randomization budget. If it grows toward the failing regions, recalibrate before collecting anything else, or new data will bake the offset in.
The five patterns at a glance:
| Pattern | Observed failure | Likely cause | First check |
|---|---|---|---|
| 1. Early-step action collapse | Degenerate or near-constant actions from the first step | Observation plumbing mismatch, such as normalization, channel order, resize filter, or units | Push matched real and simulated observations through the frozen policy and compare actions |
| 2. Trajectory tracking lag and offset | Right trajectory shape, but the end-effector is offset or lags | Controller gains, actuator latency, motor dynamics, or a loop rate mismatch | Replay joint commands with no policy in the loop and compare tracking error |
| 3. Contact transition instability | Smooth reach, then a slip, penetration, or knock-away at contact | Miscalibrated friction, compliance, or restitution | Replay a scripted open-loop grasp or push and compare object trajectories |
| 4. Visual sensitivity | Outcome flips with lighting, background, table covering, or object instance | Perception domain shift | Swap in ground-truth poses, or change one visual factor at a time |
| 5. Workspace inconsistency and spatial degradation | Reliable in one region, worse toward the edges | Bad camera-to-base extrinsics or thin training coverage | Reproject known 3D points across a workspace grid, then plot training coverage |
Real deployments rarely present one cause at a time. Poor calibration and thin edge coverage share an origin, since you set up and validated everything near the center. When a symptom fits two patterns, run both checks rather than picking the likelier one. Fix the causes in the order they contaminate each other: plumbing first, since it corrupts every observation, then tracking and calibration, since both move the gripper somewhere other than where the policy intended, then perception, and contact last.
Offline checks on your observation stack
Patterns 1 and 4 both lead here. Every check below runs offline on paired frames you can collect in an afternoon, with the robot never moving, so run them before any hardware check. Start by distrusting your eyes. The paired sim and real frames in Yardi et al. look near-identical to a person while models separate them cleanly, so a render beside a camera frame tells you nothing about what the encoder sees.
Measure instead. Collect paired sim and real frames of matched scenes, with the same object poses, camera placement, and task state. Push both through your encoder and compute normalized centroid distance between the embedding clouds after a common principal component analysis (PCA) projection. To see the gap directly, you can compute embeddings and a PCA projection with FiftyOne Brain, then color the Embeddings panel by sim or real. If you want to run this on your own frames without setting up infrastructure, you can try Voxel51 for free, no credit card required.
Then freeze the encoder and train a linear probe (a single linear layer) to predict demonstrated actions from the embeddings. Centroid distance tells you whether the encoder separates sim from real. The probe tells you whether it keeps what the policy needs. An encoder can win on the first by discarding everything, so read both as a ranking across candidates, not against a threshold.
Before blaming representations, check raw statistics on the same pairs: per-channel image means and standard deviations, depth percentiles, and joint-by-joint proprioceptive ranges. These take minutes and often explain an embedding gap outright. A channel swap, a different interpolation mode, depth in millimeters versus meters, or a normalization constant from the wrong dataset shows up as a clean offset.
Finally, check detections. If simulation gives the policy ground-truth poses or masks and hardware gives it a learned detector, those are different inputs, not noisy versions of one input. Ground truth degrades gracefully under occlusion, while an open-vocabulary detector such as Grounding DINO can drop the object or latch onto a neighbor. Run your real detector on your simulated frames and compare against ground truth. Any gap is self-inflicted and fixable without touching physics, either by training on detector output in sim or by aligning the two interfaces.
What each diagnosis buys you
Localization first tells you whether randomization is the right tool at all. Plumbing mismatches and calibration errors (the latter once reprojection confirms them) are bugs, not allocations. Randomizing around a swapped channel or a bad extrinsic spends budget covering an error you could delete, and it works just well enough to keep you from finding it.
For the rest, the symptom decides where the budget goes. Tracking failures call for system identification first, so that randomization is centered on the right values instead of hiding the offset. Contact failures call for a better contact model rather than a wider distribution over a wrong one, and they are where real data pays off most. Perception failures split two ways. If one candidate encoder places sim and real clearly closer while keeping its probe accuracy, swap to it. If every candidate keeps them far apart, the difference lives in the images, so spend on visual randomization or better rendering. Coverage failures, with flat reprojection error and failures where demonstrations are thin, are the one case where randomization is the direct answer, as long as it targets the failing regions or comes with new demonstrations there.
The gap does not close. It localizes. Each check turns a hardware failure into a specific claim about which subsystem is wrong. Most run in an afternoon, and several never touch the robot. Run them before you widen a distribution. Randomization is expensive and, worse, quiet: it covers an error rather than telling you it is there.
Frequently asked questions
The sim-to-real gap is the drop in performance when a policy trained in simulation runs on physical hardware. Its causes almost always fall into three buckets: dynamics (contact, friction, and actuator models), perception (lighting, textures, and camera parameters), and task distribution (object poses, clutter, and workspace coverage).
Sim-to-real transfer works for quadruped locomotion because locomotion forgives wrong contact assumptions: twelve actuated joints, several feet sharing the load, and a fast control loop absorb contact error. Manipulation concentrates the task into one or two contact patches with no redundancy, so the friction coefficient at the fingertip decides whether the task succeeds.
Domain randomization often narrows the sim-to-real gap by widening the training distribution until the real world falls inside it, which is why it's the default response. Randomizing everything at once wastes capacity on variation that doesn't matter and can produce a worse, overly conservative policy, so it works best after you've found which bucket the failure is in.
To diagnose a sim-to-real failure, first confirm the simulation success rate with a tight confidence interval, then match the hardware failure to one of five patterns: early-step action collapse, trajectory tracking lag, contact transition instability, visual sensitivity, or workspace degradation. Each pattern has a targeted check, and several, such as comparing raw observation statistics and embeddings on sim and real frames, run offline without moving the robot.
Source list
- Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A. and State, G., 2021. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470.
- Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. Conference on Robot Learning (CoRL 2021), PMLR 164:91-100, 2022. Code: legged_gym.
- CoRL 2021 Demos program listing, live training demonstration (demo 7) by Rudin, Hoeller, Reist, and Hutter.
- Marco Hutter ETH Learning Control Legged Robots ICRA 2022. YouTube, uploaded by the Biomechatronics and Intelligent Robotics Lab, 2024.
- Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: Rapid Motor Adaptation for Legged Robots. Robotics: Science and Systems, 2021. doi:10.15607/RSS.2021.XVII.011
- Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots. ICRA, pages 2811-2817, 2021.
- Yinuo Wang and Xiaowen Tao. LocoMamba: Vision-Driven Locomotion via End-to-End Deep Reinforcement Learning with Mamba. Advanced Engineering Informatics, 70:104230, 2026. doi:10.1016/j.aei.2025.104230
- Yunlong Song, Sangbae Kim, and Davide Scaramuzza. Learning Quadruped Locomotion Using Differentiable Simulation. Conference on Robot Learning (CoRL 2024), PMLR 270, 2025.
- Cassidy Laidlaw, Stuart Russell, and Anca Dragan. Bridging Reinforcement Learning Theory and Practice with the Effective Horizon. NeurIPS, 2023.
- Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot Learning from Randomized Simulations: A Review. Frontiers in Robotics and AI, 2022.
- Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS, pages 23-30, 2017. doi:10.1109/IROS.2017.8202133
- Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. ICRA, pages 3803-3810, 2018.
- Jack Collins, David Howard, and Jürgen Leitner. Quantifying the Reality Gap in Robotic Manipulation Tasks. ICRA, pages 6706-6712, 2019. doi:10.1109/ICRA.2019.8793591
- Andrej Karpathy. A Recipe for Training Neural Networks. Blog post, April 25, 2019.
- Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning Agile and Dynamic Motor Skills for Legged Robots. Science Robotics, 4(26):eaau5872, 2019. doi:10.1126/scirobotics.aau5872
- An Dang, Jayjun Lee, Mustafa Mukadam, X. Alice Wu, Bernadette Bucher, Manikantan Nambi, and Nima Fazeli. HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning. arXiv:2603.00446, February 2026. Project page.
- Yash Yardi, Samuel Biruduganti, and Lars Ankile. Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer. 2025.
- Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation. 2023. The dataset behind Yardi et al.
- Guangqi Jiang, Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots Pre-Train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets. 2024. The top-scoring encoder in Yardi et al.
- Stéphane Ross, Geoffrey Gordon, and J. Andrew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS, 2011.
Technical Writer
Mohamad Louai Shehab is a postdoctoral researcher in the Robotics Department at the University of Michigan.
See all articles by Mohamad Louai Shehab
Try Voxel51 for yourself
Loading related posts...
© 2026 Voxel51 All Rights Reserved
