FoundationPose

FoundationPose is a foundation model from NVIDIA for 6D object pose estimation and tracking, presented at CVPR 2024. It estimates and tracks the pose of objects it never saw in training, given either a CAD model or a small set of reference images, without fine-tuning. In March 2024 it ranked first on the BOP leaderboard for model-based pose estimation of unseen objects.

What is FoundationPose?

FoundationPose is a pose model from NVIDIA researchers Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield, presented as a highlight paper at CVPR 2024. It handles two jobs that used to need separate models: estimating an object's 6D pose in a single frame, and tracking that pose through video.
It also unifies two setups. In the model-based setup, you give it a CAD model of the object. In the model-free setup, you give it a small set of reference images of the object instead, 16 in the paper's default setup. Either way, it works on new objects at test time without fine-tuning.

Key takeaways

  • FoundationPose estimates and tracks the 6D pose of objects it never saw in training.
  • It works from a CAD model or a small set of reference images, with no per-object fine-tuning.
  • It was trained on large-scale synthetic data and ranked first on the BOP leaderboard for model-based pose estimation of unseen objects in March 2024.

How it works

For the model-free setup, FoundationPose builds a neural implicit representation of the object from the reference images, which lets it render the object from new viewpoints. That way, the same downstream steps work whether it started from a CAD model or from images. It then generates candidate poses, refines each one by comparing renderings with the observed image, and ranks the refined candidates to pick the best, using a transformer-based architecture trained with contrastive learning.
Training used large-scale synthetic data, with large language models helping to diversify object appearance. The model takes RGB-D input, so depth quality affects its estimates.
FoundationPose runs the same refine-and-rank steps from a CAD model or from reference views.

Why it matters

Most earlier pose models needed training on each specific object, which made them slow to deploy for a new part or product. FoundationPose made novel-object pose a practical starting point: give it a CAD model or a small set of reference views of the object and it runs without retraining. Its output is still only as good as its inputs. Depth noise, reflective surfaces, clutter, and occlusion all shift the estimate, which matters most in tasks like bin picking, so pose predictions are worth reviewing against the images before they become labels or grasp targets.
Without a CAD model, FoundationPose works from a set of reference views.

Frequently asked questions

Is FoundationPose open source?

Not in the usual sense. The code is public on GitHub under the NVIDIA Source Code License, which limits use to non-commercial research and evaluation. Commercial use needs a separate arrangement with NVIDIA.

What is the difference between model-based and model-free pose estimation?

Model-based methods need a 3D model of the object, usually a CAD file. Model-free methods work from a set of reference images instead, which helps when no CAD model exists.

Does FoundationPose need training for a new object?

No. It applies to new objects at test time without fine-tuning, as long as it has a CAD model or a set of reference images of the object.
black and white photo of Jesse Mostipak
Director of Growth
Jesse Mostipak is the Director of Growth at Voxel51, where the work is helping humans find and trust what the brand knows, and teaching the Google knowledge graph and the LLMs answering on their behalf to do the same. That question, how knowledge gets built inside a system, is one Jesse has been chasing for years. Earlier versions of it ran through a New York City high school science classroom, data science and machine learning, and developer relations at Kaggle, Posit (formerly RStudio), and Baseten. The answer doesn't change much depending on whether the learner is a teenager, a software engineer, or a knowledge graph. Jesse holds a Master's in Education from CUNY Hunter College. LinkedIn
See all articles by Jesse Mostipak
Last updated November 19, 2026

Building visual or physical AI?

Let's talk.