Imagine a robot that has detected a power drill in an image. The box says where the drill appears, but the gripper needs more: where the drill is in three-dimensional space and how it is rotated. FoundationPose turns an RGB-D observation and an object model into that six-degree-of-freedom pose.
Card image credit: architecture overview from the FoundationPose project page, by Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield, NVIDIA.
1. What is FoundationPose?
FoundationPose is a model for estimating and tracking the 6D pose of novel rigid objects without retraining for each object. A 6D pose contains three translation values and three rotational degrees of freedom. Together they define the rigid transformation that places an object coordinate frame inside the camera coordinate frame.
“Novel” needs a careful explanation. The released model can process an object instance that was not part of its training set, but it still needs information about that object at inference. The original paper supports a model-based setup with a textured CAD model and a model-free setup with reference views reconstructed into a neural object field. The Telekinesis Skill implements the model-based path and therefore requires a mesh.
This is not ordinary object detection. A detector can say that a rectangular image region contains a drill. FoundationPose asks which 3D rotation and translation would make the known drill geometry agree with the observed color and depth inside that region.
1.1 The Complete Registration Pipeline
RGB-D + object model → object-to-camera pose
Registration searches broadly and ranks many candidates; tracking starts near the previous pose and only refines locally.
FoundationPose begins with more information than an ordinary object detector. It receives the camera’s RGB and metric depth images, the camera intrinsics, a 3D model of the object, and a 2D region showing where the object is. The remaining question is how that known 3D model must be rotated and translated to explain the observed pixels.
For the first frame, the system cannot assume it already knows the answer. Registration samples many possible poses around the object. For each possibility, it renders the model as the camera would see it and compares that rendering with the real color and depth evidence.
Promising candidates are refined rather than accepted immediately. Refinement adjusts their rotation and translation so the rendered object lines up more closely with the observation. A scoring network then ranks the refined candidates, and the best-supported one becomes the estimated 6D pose.
Later video frames can use tracking. Instead of repeating the broad search, the system starts near the previous pose and performs local refinement. Registration finds the object pose when it is unknown; tracking follows it once a trustworthy estimate exists. The following sections unpack those two paths.
1.2 A Pose Is a Transformation Between Coordinate Frames
Attach one coordinate frame to the object’s CAD model and another to the camera. The pose returned to an application is the object-to-camera transformation
is a homogeneous transform that maps object-frame coordinates into the camera frame; is a rotation matrix; is translation; and completes the homogeneous representation.
For a mesh point , its camera-frame position is
is the point in the mesh’s object frame and is the same physical point expressed in the camera frame. This frame direction matters. Confusing object-to-camera with camera-to-object produces a plausible-looking matrix that moves the robot in the wrong direction.
The camera projects into an image pixel using its intrinsic matrix :
is the pixel coordinate, is a projective scale, and contains focal lengths and principal point. FoundationPose needs intrinsics because rendering and comparison must agree with how the real camera maps 3D geometry into pixels.
1.3 RGB-D Turns Appearance Into Geometric Evidence
Color alone can be ambiguous. A symmetric or textureless object may look similar from several orientations. Aligned depth adds the visible surface geometry in metres. Using the camera intrinsics, the system can back-project a depth pixel into a 3D observation point.
For pixel with depth , the pinhole equations are
, , and are camera-frame coordinates; and are focal lengths in pixels; and is the principal point. Incorrect depth scale or intrinsics therefore shifts the evidence against which every pose is judged.
The 2D bounding box does not itself provide pose. It tells registration which image region contains the target, helping isolate relevant depth and appearance from the rest of the scene. A tight box reduces distractors; an offset or overly loose box can corrupt initialization.
1.4 Registration Samples Global Pose Hypotheses
One observation can support many possible poses, especially before the object is aligned. FoundationPose does not ask one network to jump directly from a crop to a final transform. It begins by sampling candidate viewpoints around the object and estimating translations from the observed region and depth.
Call the candidates , where is the number of hypotheses and is one proposed object-to-camera transform. A global set is important on the first frame because the object’s orientation may be anywhere in , the space of 3D rotations.
For every candidate, the known object model can be rendered through the camera model. The result predicts what the object’s RGB-D observation should look like if that pose were correct. Pose estimation is then converted into a render-and-compare problem: align what the camera saw with what the model predicts.
This structure is one reason FoundationPose transfers to unseen objects. The network learns how to judge and correct alignment between observation and rendering rather than memorizing one output head for every object identity.
1.5 The Refinement Network Corrects Each Candidate
For a hypothesis , the model crops the observed RGB-D evidence in a pose-conditioned way and compares it with the rendered RGB-D view. The refinement network predicts a translation correction and rotation correction .
The update can be represented as
is the current pose, is the predicted rigid correction, and is the refined pose. Rotation composition is used rather than adding Euler angles because rotations live on a curved group rather than ordinary Euclidean space.
The paper normalizes translation updates relative to the object and crop so that learning remains stable across scales. Observation and update quantities are represented in a shared camera-centric frame, helping the network learn one correction rule for many objects and viewpoints.
Refinement can run more than once. The refine_iterations parameter in Telekinesis controls this repetition. More iterations may improve a nearby hypothesis, but they add latency and cannot reliably rescue a global initialization that selected the wrong object or a fundamentally incompatible viewpoint.
1.6 The Scoring Network Selects the Best Refined Pose
Refinement leaves a collection of plausible candidates. FoundationPose next assigns them scores. Each candidate’s rendered and observed evidence is encoded into an alignment feature. The model treats the candidate features as a sequence and uses self-attention so candidates can be evaluated relative to one another.
If the resulting scores are , the selected pose is
is the predicted alignment quality for candidate , is the index of the highest-scoring candidate, and the final pose is . The score is a ranking signal, not automatically a calibrated probability that the pose is safe for manipulation.
No positional encoding is applied to the candidate sequence, making the comparison insensitive to candidate ordering. That is appropriate because hypothesis number three has no inherent semantic precedence over hypothesis number eight.
1.7 The Two Training Losses Teach Correction and Ranking
FoundationPose contains separate refinement and scoring networks, so it uses different learning objectives for their different jobs.
The refinement loss is
is the correction loss; and are predicted translation and rotation updates; and are ground-truth updates; and and balance translation and rotation terms. The distances penalize incorrect corrections.
The ranking network uses a pose-conditioned triplet-style loss:
identifies a candidate sufficiently close to ground truth; identifies a competing candidate; is the scorer output; and is the desired score margin. Loss appears when the positive candidate does not outrank the negative by at least .
The total ranking loss sums these comparisons over meaningful positive–negative pairs. Positives are selected using pose error, including rotation geodesic distance and the ADD family of model-point metrics. Very poor candidates are not treated as interchangeable positives merely because both are far away.
Together, the losses divide responsibility cleanly: refinement learns how to move a candidate toward truth, while ranking learns which refined candidate should be trusted most.
1.8 What FoundationPose Was Trained On
The paper creates large-scale synthetic RGB-D training scenes from object assets in Google Scanned Objects and Objaverse. The released model documentation describes more than 40,000 Objaverse-LVIS objects spanning 1,156 categories, alongside roughly 1,000 GSO objects. NVIDIA Isaac Sim and Omniverse provide physically plausible placement, path-traced rendering, depth, segmentation, boxes, camera poses, and exact object poses.
Domain randomization varies lighting, materials, distractors, viewpoints, occlusion, and backgrounds. The authors also use language-model-assisted scene composition and diffusion-based texture augmentation in parts of the research pipeline. The official open release notes that legal restrictions prevent release of diffusion-augmented training data and weights, so its behaviour can differ slightly from the fullest paper configuration.
Synthetic supervision is particularly valuable here because exact 6D annotations, depth, visibility, and rendered comparisons are available automatically. The sim-to-real gap remains real: sensor noise, transparent parts, proprietary surface finishes, damaged meshes, and calibration errors can all differ from the renderer.
1.9 Tracking Starts From the Previous Pose
Global registration is expensive because it must search many orientations. Once frame has a reliable pose, frame usually begins nearby. Tracking uses the previous state to initialize refinement instead of sampling the full global hypothesis set.
Telekinesis exposes this distinction explicitly. Provide bbox to register, or provide a prior call’s tracking_pose as previous_pose to track. Exactly one is required. The returned application pose and opaque tracking_pose are not interchangeable: use pose as the mesh’s object-to-camera transform, and feed tracking_pose back only as continuation state.
Tracking assumes limited inter-frame motion. Low capture rate, rapid object motion, occlusion, or one bad update can cause drift. Re-register with a fresh box when validation fails rather than repeatedly refining a stale state.
1.10 Symmetry Makes Some Poses Indistinguishable
A plain cylinder rotated around its long axis can produce the same visible geometry. A cube may admit several equivalent orientations. If an object is symmetric, a numerically different rotation is not necessarily physically wrong.
For mesh points , the ADD pose error is
is a set of model points, its size, the estimated pose, the reference pose, and a homogeneous model point. For symmetric objects, ADD-S replaces direct correspondence with the nearest transformed model point so equivalent symmetries are not unfairly penalized.
Robotic evaluation must follow the task. A rotation that is irrelevant for grasping a round spacer may be critical for inserting a keyed connector.
Interactive pose ambiguity
Compare a pose hypothesis with symmetric observations
1.11 Engineering Strengths and Limitations
FoundationPose’s main strength is instance transfer: a usable CAD mesh allows one trained system to handle a new rigid object without collecting a new per-object pose dataset. RGB-D comparison supplies both appearance and geometry, while registration and tracking cover first-frame initialization and efficient continuation.
Its assumptions are equally important:
- The target must be rigid and represented at the correct scale in the mesh.
- RGB and depth must be spatially aligned and synchronized.
- Camera intrinsics and depth scale must be accurate.
- Registration needs a reasonably tight box around the correct instance.
- Heavy occlusion removes evidence needed for comparison.
- Similar or symmetric poses may be fundamentally ambiguous.
- A camera-frame pose is not yet a robot-frame grasp. It must pass through calibrated frame transforms, grasp selection, reachability, and collision checking.
Treat the result as a geometric estimate with measurable uncertainty, not as permission to move a robot.
2. How to Use FoundationPose in the Telekinesis Agentic OS
Telekinesis provides FoundationPose through the Vitreous Skill estimate_pose_using_foundation_pose. The first call registers from a box. The second demonstrates the intended tracking contract by feeding back the returned continuation state.
from telekinesis import vitreous
# First frame: search broadly inside the detected object box.
pose, tracking_pose = vitreous.estimate_pose_using_foundation_pose(
rgb_image=rgb_image,
depth_image=depth_image,
mesh=mesh,
camera_calibration=camera_calibration,
bbox=[280, 200, 360, 280], # [xmin, ymin, xmax, ymax]
refine_iterations=2,
)
print("Object-to-camera transform:")
print(pose.data)
# Later frame: refine from the previous tracking state.
next_pose, next_tracking_pose = (
vitreous.estimate_pose_using_foundation_pose(
rgb_image=next_rgb_image,
depth_image=next_depth_image,
mesh=mesh,
camera_calibration=camera_calibration,
previous_pose=tracking_pose,
refine_iterations=2,
)
)
The RGB and depth images must share image geometry, and depth must be metric. The mesh must use the physical scale expected by the scene. Use the returned pose in your application. Treat tracking_pose as opaque state: pass it back to the Skill, but do not interpret it as the public mesh-frame transform.
3. Benchmarking
3.1 Model-Free Pose Estimation
The FoundationPose paper tests both model-free operation from reference views and model-based operation from a CAD mesh. On YCB-Video, all methods receive 16 reference images and are scored by area under the ADD and symmetry-aware ADD-S curves.
| Method | Fine-tune free | ADD-S AUC ↑ | ADD AUC ↑ |
|---|---|---|---|
| PREDATOR | Yes | 71.0 | 24.3 |
| LoFTR | Yes | 52.5 | 26.2 |
| FS6D-DPM | No | 88.4 | 42.1 |
| FoundationPose | Yes | 97.4 | 91.5 |
On LINEMOD, ADD-0.1d counts a pose as correct when its average transformed model-point error is below 10% of the object’s diameter.
| Method | Input | Fine-tune free | Reference images | Mean ADD-0.1d ↑ |
|---|---|---|---|---|
| OnePose | RGB | No | 200 | 63.6 |
| OnePose++ | RGB | No | 200 | 76.9 |
| LatentFusion | RGB-D | No | 16 | 87.1 |
| FS6D | RGB-D | Yes | 16 | 88.9 |
| FS6D + ICP | RGB-D | Yes | 16 | 91.5 |
| FoundationPose | RGB-D | Yes | 16 | 99.9 |
3.2 Model-Based Pose Estimation on BOP Datasets
The BOP average-recall score combines VSD, MSSD, and MSPD, covering visible-surface alignment, symmetry-aware 3D error, and projection error.
| Method | Novel objects | LM-O AR | T-LESS AR | YCB-V AR | Mean AR |
|---|---|---|---|---|---|
| SurfEmb + ICP | No | 75.8 | 82.8 | 80.6 | 79.7 |
| MegaPose-RGBD | Yes | 58.3 | 54.3 | 63.3 | 58.6 |
| OVE6D | Yes | 49.6 | 52.3 | — | — |
| GCPose | Yes | 65.2 | 67.9 | — | — |
| FoundationPose | Yes | 78.8 | 83.0 | 88.0 | 83.3 |
3.3 Pose Tracking
YCBInEOAT contains dynamic robot manipulation, occlusion, and abrupt viewpoint changes. The daggered FoundationPose result initializes tracking with the model’s own estimator instead of a ground-truth pose.
| Tracker | Novel objects | Initial pose | ADD-S AUC ↑ | ADD AUC ↑ |
|---|---|---|---|---|
| se(3)-TrackNet | No | Ground truth | 95.53 | 92.66 |
| RGF | Yes | Ground truth | 39.90 | 29.98 |
| BundleTrack | Yes | Ground truth | 92.53 | 87.34 |
| BundleSDF | Yes | Ground truth | 93.77 | 86.95 |
| Wüthrich et al. | Yes | Ground truth | 89.18 | 78.28 |
| FoundationPose | Yes | Ground truth | 96.42 | 93.09 |
| FoundationPose, full pipeline | Yes | Estimated | 96.40 | 93.22 |
3.4 What to Measure on a Robot
Begin with ADD or symmetry-aware ADD-S, rotation error in degrees, and translation error in millimetres. Report success against a physical tolerance, not only a dataset average. Separate initial registration from tracking because they have different runtime and failure behaviour.
Then evaluate the pipeline conditions that matter: box quality, depth noise, object distance, visibility, symmetry, motion between frames, and time since last registration. Measure registration latency, tracking latency, drift, re-registration frequency, and catastrophic failures in which the mesh aligns to the wrong instance.
Finally, measure downstream task error. Transform the pose into the robot base frame, execute collision-checked dry runs, and report grasp or insertion success. A low ADD value can still be unusable when the mesh origin does not match the grasp convention, while a symmetry-equivalent pose can be perfectly adequate.
4. Where to Go Next
5. References
- Wen, B., Yang, W., Kautz, J., and Birchfield, S. “FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects”, CVPR 2024.
- Calli, B. et al. “Benchmarking in Manipulation Research: The YCB Object and Model Set”, IEEE Robotics & Automation Magazine, 2015.
- Hinterstoisser, S. et al. “Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes”, ACCV 2012.
- Telekinesis. “Estimate Pose Using FoundationPose”, Vitreous Skill documentation.



