3D Perception · Vitreous

What is FoundationPose?

FoundationPose solves the problem of locating a previously unseen rigid object in 3D when its shape is known. Learn how pose hypotheses are rendered, refined, ranked, and tracked, which losses train the model, and how to use it in the Telekinesis Agentic OS.

Imagine a robot that has detected a power drill in an image. The box says where the drill appears, but the gripper needs more: where the drill is in three-dimensional space and how it is rotated. FoundationPose turns an RGB-D observation and an object model into that six-degree-of-freedom pose.

Watch the estimated object frame remain aligned as objects and cameras move. The tutorial below explains how registration, refinement, scoring, and tracking produce that alignment.

Card image credit: architecture overview from the FoundationPose project page, by Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield, NVIDIA.

1. What is FoundationPose?

FoundationPose is a model for estimating and tracking the 6D pose of novel rigid objects without retraining for each object. A 6D pose contains three translation values and three rotational degrees of freedom. Together they define the rigid transformation that places an object coordinate frame inside the camera coordinate frame.

“Novel” needs a careful explanation. The released model can process an object instance that was not part of its training set, but it still needs information about that object at inference. The original paper supports a model-based setup with a textured CAD model and a model-free setup with reference views reconstructed into a neural object field. The Telekinesis Skill implements the model-based path and therefore requires a mesh.

This is not ordinary object detection. A detector can say that a rectangular image region contains a drill. FoundationPose asks which 3D rotation and translation would make the known drill geometry agree with the observed color and depth inside that region.

1.1 The Complete Registration Pipeline

RGB-D + object model → object-to-camera pose

RGB-D observationAligned color and metric depth
Mesh + intrinsicsObject geometry and camera model
Pose hypothesesSample candidate viewpoints around the object
Render and compareContrast observed RGB-D with each rendered pose
Pose refinement networkPredict translation and rotation corrections
Pose scoring networkRank refined hypotheses using global context
6D object poseRigid object-to-camera transformation

Registration searches broadly and ranks many candidates; tracking starts near the previous pose and only refines locally.

In its model-based setup, FoundationPose initializes global pose hypotheses, renders the known object from those poses, refines observation-to-render alignment, and selects the highest-scoring result. Adapted as an explanatory diagram fromWen et al., “FoundationPose”.

FoundationPose begins with more information than an ordinary object detector. It receives the camera’s RGB and metric depth images, the camera intrinsics, a 3D model of the object, and a 2D region showing where the object is. The remaining question is how that known 3D model must be rotated and translated to explain the observed pixels.

For the first frame, the system cannot assume it already knows the answer. Registration samples many possible poses around the object. For each possibility, it renders the model as the camera would see it and compares that rendering with the real color and depth evidence.

Promising candidates are refined rather than accepted immediately. Refinement adjusts their rotation and translation so the rendered object lines up more closely with the observation. A scoring network then ranks the refined candidates, and the best-supported one becomes the estimated 6D pose.

Later video frames can use tracking. Instead of repeating the broad search, the system starts near the previous pose and performs local refinement. Registration finds the object pose when it is unknown; tracking follows it once a trustworthy estimate exists. The following sections unpack those two paths.

1.2 A Pose Is a Transformation Between Coordinate Frames

Attach one coordinate frame to the object’s CAD model and another to the camera. The pose returned to an application is the object-to-camera transformation

CTO=[Rt0T1].{}^{C}\mathbf T_O= \begin{bmatrix} \mathbf R & \mathbf t\\ \mathbf 0^{\mathsf T} & 1 \end{bmatrix}.

CTO{}^{C}\mathbf T_O is a 4×44\times4 homogeneous transform that maps object-frame coordinates into the camera frame; R∈SO(3)\mathbf R\in SO(3) is a 3×33\times3 rotation matrix; t∈R3\mathbf t\in\mathbb R^3 is translation; and 0T\mathbf 0^{\mathsf T} completes the homogeneous representation.

For a mesh point pO=[xO,yO,zO]T\mathbf p_O=[x_O,y_O,z_O]^{\mathsf T}, its camera-frame position is

pC=RpO+t.\mathbf p_C=\mathbf R\mathbf p_O+\mathbf t.

pO\mathbf p_O is the point in the mesh’s object frame and pC\mathbf p_C is the same physical point expressed in the camera frame. This frame direction matters. Confusing object-to-camera with camera-to-object produces a plausible-looking matrix that moves the robot in the wrong direction.

The camera projects pC=[X,Y,Z]T\mathbf p_C=[X,Y,Z]^{\mathsf T} into an image pixel using its intrinsic matrix K\mathbf K:

s[uv1]=K[XYZ].s \begin{bmatrix}u\\v\\1\end{bmatrix} = \mathbf K \begin{bmatrix}X\\Y\\Z\end{bmatrix}.

(u,v)(u,v) is the pixel coordinate, ss is a projective scale, and K\mathbf K contains focal lengths and principal point. FoundationPose needs intrinsics because rendering and comparison must agree with how the real camera maps 3D geometry into pixels.

1.3 RGB-D Turns Appearance Into Geometric Evidence

Color alone can be ambiguous. A symmetric or textureless object may look similar from several orientations. Aligned depth adds the visible surface geometry in metres. Using the camera intrinsics, the system can back-project a depth pixel into a 3D observation point.

For pixel (u,v)(u,v) with depth ZZ, the pinhole equations are

X=(u−cx)Zfx,Y=(v−cy)Zfy.X=\frac{(u-c_x)Z}{f_x},\qquad Y=\frac{(v-c_y)Z}{f_y}.

XX, YY, and ZZ are camera-frame coordinates; fxf_x and fyf_y are focal lengths in pixels; and (cx,cy)(c_x,c_y) is the principal point. Incorrect depth scale or intrinsics therefore shifts the evidence against which every pose is judged.

The 2D bounding box does not itself provide pose. It tells registration which image region contains the target, helping isolate relevant depth and appearance from the rest of the scene. A tight box reduces distractors; an offset or overly loose box can corrupt initialization.

1.4 Registration Samples Global Pose Hypotheses

One observation can support many possible poses, especially before the object is aligned. FoundationPose does not ask one network to jump directly from a crop to a final transform. It begins by sampling candidate viewpoints around the object and estimating translations from the observed region and depth.

Call the candidates {Ti}i=1K\{\mathbf T_i\}_{i=1}^{K}, where KK is the number of hypotheses and Ti\mathbf T_i is one proposed object-to-camera transform. A global set is important on the first frame because the object’s orientation may be anywhere in SO(3)SO(3), the space of 3D rotations.

For every candidate, the known object model can be rendered through the camera model. The result predicts what the object’s RGB-D observation should look like if that pose were correct. Pose estimation is then converted into a render-and-compare problem: align what the camera saw with what the model predicts.

This structure is one reason FoundationPose transfers to unseen objects. The network learns how to judge and correct alignment between observation and rendering rather than memorizing one output head for every object identity.

1.5 The Refinement Network Corrects Each Candidate

For a hypothesis Ti\mathbf T_i, the model crops the observed RGB-D evidence in a pose-conditioned way and compares it with the rendered RGB-D view. The refinement network predicts a translation correction Δt\Delta\mathbf t and rotation correction ΔR\Delta\mathbf R.

The update can be represented as

Ti′=ΔTiTi.\mathbf T_i' = \Delta\mathbf T_i\mathbf T_i.

Ti\mathbf T_i is the current pose, ΔTi\Delta\mathbf T_i is the predicted rigid correction, and Ti′\mathbf T_i' is the refined pose. Rotation composition is used rather than adding Euler angles because rotations live on a curved group rather than ordinary Euclidean space.

The paper normalizes translation updates relative to the object and crop so that learning remains stable across scales. Observation and update quantities are represented in a shared camera-centric frame, helping the network learn one correction rule for many objects and viewpoints.

Refinement can run more than once. The refine_iterations parameter in Telekinesis controls this repetition. More iterations may improve a nearby hypothesis, but they add latency and cannot reliably rescue a global initialization that selected the wrong object or a fundamentally incompatible viewpoint.

1.6 The Scoring Network Selects the Best Refined Pose

Refinement leaves a collection of plausible candidates. FoundationPose next assigns them scores. Each candidate’s rendered and observed evidence is encoded into an alignment feature. The model treats the KK candidate features as a sequence and uses self-attention so candidates can be evaluated relative to one another.

If the resulting scores are S=[S1,…,SK]\mathbf S=[S_1,\ldots,S_K], the selected pose is

i∗=argmax⁡i∈{1,…,K}Si.i^*=\operatorname*{argmax}_{i\in\{1,\ldots,K\}}S_i.

SiS_i is the predicted alignment quality for candidate ii, i∗i^* is the index of the highest-scoring candidate, and the final pose is Ti∗′\mathbf T_{i^*}'. The score is a ranking signal, not automatically a calibrated probability that the pose is safe for manipulation.

No positional encoding is applied to the candidate sequence, making the comparison insensitive to candidate ordering. That is appropriate because hypothesis number three has no inherent semantic precedence over hypothesis number eight.

1.7 The Two Training Losses Teach Correction and Ranking

FoundationPose contains separate refinement and scoring networks, so it uses different learning objectives for their different jobs.

The refinement loss is

Lrefine=w1∥Δt−Δtˉ∥2+w2∥ΔR−ΔRˉ∥2.\mathcal L_{refine}= w_1\left\|\Delta\mathbf t-\Delta\bar{\mathbf t}\right\|_2 +w_2\left\|\Delta\mathbf R-\Delta\bar{\mathbf R}\right\|_2.

Lrefine\mathcal L_{refine} is the correction loss; Δt\Delta\mathbf t and ΔR\Delta\mathbf R are predicted translation and rotation updates; Δtˉ\Delta\bar{\mathbf t} and ΔRˉ\Delta\bar{\mathbf R} are ground-truth updates; and w1w_1 and w2w_2 balance translation and rotation terms. The L2L_2 distances penalize incorrect corrections.

The ranking network uses a pose-conditioned triplet-style loss:

L(i+,i−)=max⁡(S(i−)−S(i+)+α,0).\mathcal L(i^+,i^-)= \max\left(S(i^-)-S(i^+)+\alpha,0\right).

i+i^+ identifies a candidate sufficiently close to ground truth; i−i^- identifies a competing candidate; S(⋅)S(\cdot) is the scorer output; and α\alpha is the desired score margin. Loss appears when the positive candidate does not outrank the negative by at least α\alpha.

The total ranking loss sums these comparisons over meaningful positive–negative pairs. Positives are selected using pose error, including rotation geodesic distance and the ADD family of model-point metrics. Very poor candidates are not treated as interchangeable positives merely because both are far away.

Together, the losses divide responsibility cleanly: refinement learns how to move a candidate toward truth, while ranking learns which refined candidate should be trusted most.

1.8 What FoundationPose Was Trained On

The paper creates large-scale synthetic RGB-D training scenes from object assets in Google Scanned Objects and Objaverse. The released model documentation describes more than 40,000 Objaverse-LVIS objects spanning 1,156 categories, alongside roughly 1,000 GSO objects. NVIDIA Isaac Sim and Omniverse provide physically plausible placement, path-traced rendering, depth, segmentation, boxes, camera poses, and exact object poses.

Domain randomization varies lighting, materials, distractors, viewpoints, occlusion, and backgrounds. The authors also use language-model-assisted scene composition and diffusion-based texture augmentation in parts of the research pipeline. The official open release notes that legal restrictions prevent release of diffusion-augmented training data and weights, so its behaviour can differ slightly from the fullest paper configuration.

Synthetic supervision is particularly valuable here because exact 6D annotations, depth, visibility, and rendered comparisons are available automatically. The sim-to-real gap remains real: sensor noise, transparent parts, proprietary surface finishes, damaged meshes, and calibration errors can all differ from the renderer.

1.9 Tracking Starts From the Previous Pose

Global registration is expensive because it must search many orientations. Once frame tt has a reliable pose, frame t+1t+1 usually begins nearby. Tracking uses the previous state to initialize refinement instead of sampling the full global hypothesis set.

Telekinesis exposes this distinction explicitly. Provide bbox to register, or provide a prior call’s tracking_pose as previous_pose to track. Exactly one is required. The returned application pose and opaque tracking_pose are not interchangeable: use pose as the mesh’s object-to-camera transform, and feed tracking_pose back only as continuation state.

Tracking assumes limited inter-frame motion. Low capture rate, rapid object motion, occlusion, or one bad update can cause drift. Re-register with a fresh box when validation fails rather than repeatedly refining a stale state.

1.10 Symmetry Makes Some Poses Indistinguishable

A plain cylinder rotated around its long axis can produce the same visible geometry. A cube may admit several equivalent orientations. If an object is symmetric, a numerically different rotation is not necessarily physically wrong.

For mesh points M\mathcal M, the ADD pose error is

ADD=1∣M∣∑x∈M∥Tx−Tˉx∥2.ADD=\frac{1}{|\mathcal M|} \sum_{\mathbf x\in\mathcal M} \left\|\mathbf T\mathbf x-\bar{\mathbf T}\mathbf x\right\|_2.

M\mathcal M is a set of model points, ∣M∣|\mathcal M| its size, T\mathbf T the estimated pose, Tˉ\bar{\mathbf T} the reference pose, and x\mathbf x a homogeneous model point. For symmetric objects, ADD-S replaces direct correspondence with the nearest transformed model point so equivalent symmetries are not unfairly penalized.

Robotic evaluation must follow the task. A rotation that is irrelevant for grasping a round spacer may be critical for inserting a keyed connector.

Interactive pose ambiguity

Compare a pose hypothesis with symmetric observations

—

The green outline is the observation; the red outline is the pose hypothesis. A four-fold object has the same silhouette after every 90° rotation.

1.11 Engineering Strengths and Limitations

FoundationPose’s main strength is instance transfer: a usable CAD mesh allows one trained system to handle a new rigid object without collecting a new per-object pose dataset. RGB-D comparison supplies both appearance and geometry, while registration and tracking cover first-frame initialization and efficient continuation.

Its assumptions are equally important:

  • The target must be rigid and represented at the correct scale in the mesh.
  • RGB and depth must be spatially aligned and synchronized.
  • Camera intrinsics and depth scale must be accurate.
  • Registration needs a reasonably tight box around the correct instance.
  • Heavy occlusion removes evidence needed for comparison.
  • Similar or symmetric poses may be fundamentally ambiguous.
  • A camera-frame pose is not yet a robot-frame grasp. It must pass through calibrated frame transforms, grasp selection, reachability, and collision checking.

Treat the result as a geometric estimate with measurable uncertainty, not as permission to move a robot.

2. How to Use FoundationPose in the Telekinesis Agentic OS

Telekinesis provides FoundationPose through the Vitreous Skill estimate_pose_using_foundation_pose. The first call registers from a box. The second demonstrates the intended tracking contract by feeding back the returned continuation state.

from telekinesis import vitreous

# First frame: search broadly inside the detected object box.
pose, tracking_pose = vitreous.estimate_pose_using_foundation_pose(
    rgb_image=rgb_image,
    depth_image=depth_image,
    mesh=mesh,
    camera_calibration=camera_calibration,
    bbox=[280, 200, 360, 280],  # [xmin, ymin, xmax, ymax]
    refine_iterations=2,
)

print("Object-to-camera transform:")
print(pose.data)

# Later frame: refine from the previous tracking state.
next_pose, next_tracking_pose = (
    vitreous.estimate_pose_using_foundation_pose(
        rgb_image=next_rgb_image,
        depth_image=next_depth_image,
        mesh=mesh,
        camera_calibration=camera_calibration,
        previous_pose=tracking_pose,
        refine_iterations=2,
    )
)

The RGB and depth images must share image geometry, and depth must be metric. The mesh must use the physical scale expected by the scene. Use the returned pose in your application. Treat tracking_pose as opaque state: pass it back to the Skill, but do not interpret it as the public mesh-frame transform.

Try it out

Estimate 6D object pose with Vitreous

See the complete registration and tracking contract, required data types, parameters, and return schema.

Open FoundationPose docs

Runnable example

Run the FoundationPose example

Open the exact Python example for registering a pose and carrying it into a tracking call.

View Python example

3. Benchmarking

3.1 Model-Free Pose Estimation

The FoundationPose paper tests both model-free operation from reference views and model-based operation from a CAD mesh. On YCB-Video, all methods receive 16 reference images and are scored by area under the ADD and symmetry-aware ADD-S curves.

MethodFine-tune freeADD-S AUC ↑ADD AUC ↑
PREDATORYes71.024.3
LoFTRYes52.526.2
FS6D-DPMNo88.442.1
FoundationPoseYes97.491.5

On LINEMOD, ADD-0.1d counts a pose as correct when its average transformed model-point error is below 10% of the object’s diameter.

MethodInputFine-tune freeReference imagesMean ADD-0.1d ↑
OnePoseRGBNo20063.6
OnePose++RGBNo20076.9
LatentFusionRGB-DNo1687.1
FS6DRGB-DYes1688.9
FS6D + ICPRGB-DYes1691.5
FoundationPoseRGB-DYes1699.9

3.2 Model-Based Pose Estimation on BOP Datasets

The BOP average-recall score combines VSD, MSSD, and MSPD, covering visible-surface alignment, symmetry-aware 3D error, and projection error.

MethodNovel objectsLM-O ART-LESS ARYCB-V ARMean AR
SurfEmb + ICPNo75.882.880.679.7
MegaPose-RGBDYes58.354.363.358.6
OVE6DYes49.652.3——
GCPoseYes65.267.9——
FoundationPoseYes78.883.088.083.3

3.3 Pose Tracking

YCBInEOAT contains dynamic robot manipulation, occlusion, and abrupt viewpoint changes. The daggered FoundationPose result initializes tracking with the model’s own estimator instead of a ground-truth pose.

TrackerNovel objectsInitial poseADD-S AUC ↑ADD AUC ↑
se(3)-TrackNetNoGround truth95.5392.66
RGFYesGround truth39.9029.98
BundleTrackYesGround truth92.5387.34
BundleSDFYesGround truth93.7786.95
Wüthrich et al.YesGround truth89.1878.28
FoundationPoseYesGround truth96.4293.09
FoundationPose, full pipelineYesEstimated96.4093.22

3.4 What to Measure on a Robot

Begin with ADD or symmetry-aware ADD-S, rotation error in degrees, and translation error in millimetres. Report success against a physical tolerance, not only a dataset average. Separate initial registration from tracking because they have different runtime and failure behaviour.

Then evaluate the pipeline conditions that matter: box quality, depth noise, object distance, visibility, symmetry, motion between frames, and time since last registration. Measure registration latency, tracking latency, drift, re-registration frequency, and catastrophic failures in which the mesh aligns to the wrong instance.

Finally, measure downstream task error. Transform the pose into the robot base frame, execute collision-checked dry runs, and report grasp or insertion success. A low ADD value can still be unusable when the mesh origin does not match the grasp convention, while a symmetry-equivalent pose can be perfectly adequate.

4. Where to Go Next

5. References