3D Perception · Vitreous

What is Fast-FoundationStereo?

Fast-FoundationStereo solves the problem of recovering metric scene depth from two calibrated camera images quickly enough for real-time perception. Learn stereo geometry, its feature–cost–refinement architecture, its training losses, and how to use it in the Telekinesis Agentic OS.

Imagine a robot approaching a workbench with two cameras mounted a few centimetres apart. A bolt appears slightly farther to the left in one image than in the other. That small shift is not a nuisance: together with the camera geometry, it reveals how far the bolt is from the robot.

Watch Fast-FoundationStereo turn left and right camera views into dense scene depth. The tutorial below explains what the model must infer between those inputs and the final depth map.

Card image credit: architecture overview from the Fast-FoundationStereo project page, by Bowen Wen, Shaurya Dewan, and Stan Birchfield, NVIDIA.

1. What is Fast-FoundationStereo?

Fast-FoundationStereo is a real-time neural stereo-matching model that estimates disparity from a rectified pair of images and generalizes to scenes not used for task-specific fine-tuning. Disparity is the horizontal displacement of the same scene point between the left and right images. Once disparity is known, calibrated stereo geometry converts it into metric depth.

This distinction is the first useful mental model. The network does not directly discover metres from pixels. It solves a correspondence problem: for each useful point in the left image, where is the matching point in the right image? The focal length and physical distance between the cameras then give that correspondence a metric scale.

Fast-FoundationStereo builds on FoundationStereo, which was designed for strong zero-shot stereo matching but was expensive to run. The newer model keeps the same broad pipeline—feature extraction, cost filtering, and iterative disparity refinement—while accelerating each part through knowledge distillation, neural architecture search, and structured pruning.

1.1 The Complete Path From Two Images to Depth

Rectified stereo pair → disparity → metric depth

Left imageReference view with known intrinsics
Right imageRectified second camera view
Distilled feature backboneEfficient multi-scale features for both views
Matching cost volumeCompare possible horizontal correspondences
Searched cost filteringResolve ambiguous matches with spatial context
Pruned ConvGRU refinementIteratively improve the disparity field
Metric depth imageDepth from focal length × baseline ÷ disparity

The network predicts disparity; calibrated camera geometry—not the neural network alone—turns that disparity into meters.

Fast-FoundationStereo retains the three-stage stereo pipeline of feature extraction, cost filtering, and iterative disparity refinement while accelerating each stage through distillation, blockwise architecture search, and structured pruning. Adapted as an explanatory diagram fromWen et al., “Fast-FoundationStereo”.

The pipeline starts with two cameras viewing the same scene from slightly different horizontal positions. Before the model sees the images, calibration and rectification place matching scene points on the same image row. This turns a difficult search across the whole second image into a search left or right along one line.

Both images then pass through the same feature network. The network replaces raw color patches with learned descriptions that are easier to compare across the two views. For each location in the left image, a cost volume records how well it matches candidate locations in the right image at different horizontal shifts.

Some matches are obvious; others are ambiguous because a surface is textureless, repeated, reflective, or partly hidden. The model uses surrounding image context to filter those possibilities, chooses an initial shift for every pixel, and improves that disparity estimate through several refinement steps.

Only then does camera geometry convert disparity into metric depth. Large left–right shifts usually indicate nearby points; small shifts indicate distant points. Keeping learned matching separate from this final geometric conversion makes the assumptions clear, so we begin the deep dive with rectification and disparity.

1.2 Stereo Geometry Turns Image Shift Into Distance

Two cameras see the same 3D point from different horizontal positions. In raw images, lens distortion and camera rotation can place the corresponding pixels almost anywhere. Stereo rectification warps both images so that corresponding points lie on the same image row. Matching can then search horizontally rather than across an entire image.

Let a scene point appear at horizontal coordinate uLu_L in the left image and uRu_R in the right image. Its disparity is

d=uL−uR.d=u_L-u_R.

Here, dd is disparity in pixels, while uLu_L and uRu_R are horizontal pixel coordinates in the rectified images. For a conventional stereo arrangement, nearby points move farther between views and therefore have larger disparity.

Depth follows from similar triangles:

Z=fxBd.Z=\frac{f_xB}{d}.

ZZ is depth along the camera’s forward axis in metres, fxf_x is the horizontal focal length expressed in pixels, BB is the stereo baseline in metres, and dd is disparity in pixels. Because the pixel units cancel, the result inherits the baseline’s unit.

Suppose fx=600f_x=600 pixels, B=0.05B=0.05 metres, and d=20d=20 pixels. The estimated depth is 600×0.05/20=1.5600\times0.05/20=1.5 metres. If the measured baseline is wrong by ten percent, the resulting depth scale is wrong by ten percent too. No learned model can repair incorrect calibration supplied after matching.

The inverse relationship also explains the range limit. At long distances, disparity becomes very small. A one-pixel disparity error then creates a large depth error. Stereo is strongest where the baseline, focal length, image resolution, and working distance produce measurable displacement.

Interactive stereo geometry

Disparity turns horizontal image shift into depth

—
left imageright image

Large disparity means the corresponding pixels are far apart and the point is close. Small disparity means the point is far away, where one-pixel errors matter much more.

The relationship is Z = fB/d for rectified cameras.

1.3 The Backbone Extracts Features From Both Views

Comparing raw RGB pixels is fragile. A surface can look slightly different between cameras because of exposure, reflections, occlusion, or sensor noise. Fast-FoundationStereo first turns each image into a pyramid of learned feature maps. Fine maps preserve edges and small objects; coarser maps carry wider context.

FoundationStereo obtained rich priors from a hybrid backbone that combined stereo features with a large monocular vision foundation model. That helped generalization but cost too much for real-time use. Fast-FoundationStereo distils those hybrid teacher features into one efficient student backbone.

For feature level ii, the distillation objective can be written as

Lfeat=∑i∥fstudent(i)−fˉteacher(i)∥22.\mathcal L_{feat}=\sum_i \left\|f^{(i)}_{student}-\bar f^{(i)}_{teacher}\right\|_2^2.

Lfeat\mathcal L_{feat} is the feature-distillation loss; fstudent(i)f^{(i)}_{student} is the student’s feature map at pyramid level ii; fˉteacher(i)\bar f^{(i)}_{teacher} is the frozen teacher feature at the same level; and ∥⋅∥22\lVert\cdot\rVert_2^2 sums squared element-wise differences. A projection aligns channel dimensions when the teacher and student widths differ.

The student is therefore not merely trained to reproduce the teacher’s final disparity. It is taught to construct similar intermediate evidence at several spatial scales.

1.4 A Cost Volume Stores Possible Correspondences

The backbone has described both images, but it has not yet selected matches. The next stage creates a cost volume. For every left-image location, it compares the left feature with right-image features at a range of candidate disparities.

Conceptually, a matching score can be pictured as

C(x,y,d)=sim⁡(fL(x,y),fR(x−d,y)).C(x,y,d)=\operatorname{sim} \left(f_L(x,y),f_R(x-d,y)\right).

C(x,y,d)C(x,y,d) is the evidence for disparity dd at left-image pixel (x,y)(x,y); fLf_L and fRf_R are left and right feature vectors; and sim⁡\operatorname{sim} is a learned or correlation-based similarity operation. Rectification is visible in the equation: the row yy remains fixed while only the horizontal coordinate changes.

The max_disp setting limits the tested values of dd. A larger range covers closer objects or wider baselines but increases computation and memory. If the true disparity falls outside the range, the correct match is unavailable regardless of how good the later network is.

A raw cost volume remains ambiguous. A blank wall offers many nearly identical matches. Repetitive shelves create several plausible displacements. Occluded pixels may have no counterpart at all. This is why stereo networks need cost filtering rather than independently choosing the best number at every pixel.

1.5 Cost Filtering Uses Context to Resolve Ambiguity

Cost filtering lets neighbouring evidence and long-range scene structure influence each match. FoundationStereo used a powerful combination of 3D processing and transformer reasoning over the cost volume. Fast-FoundationStereo seeks cheaper blocks that preserve as much of that behaviour as possible.

Instead of searching complete networks from scratch, the paper divides the filtering network into local blocks. Candidate replacements for each block are trained to mimic the corresponding frozen teacher block:

Lblock=∥Bic(fi−1)−Bˉi(fi−1)∥22.\mathcal L_{block}= \left\|B_i^c(f_{i-1})-\bar B_i(f_{i-1})\right\|_2^2.

BicB_i^c is candidate cc for block position ii; Bˉi\bar B_i is the teacher block; fi−1f_{i-1} is the teacher feature entering that position; and Lblock\mathcal L_{block} is the squared feature-reconstruction loss. For the last block, which predicts initial disparity rather than an intermediate feature, the paper uses smooth L1L_1 supervision against ground truth.

Each candidate is then inserted into the full teacher pipeline and measured for both error and runtime. Integer linear programming selects one candidate per position under a latency budget. This blockwise approach makes an otherwise enormous combinatorial search practical and produces several speed–accuracy variants rather than one universal network.

1.6 A Pruned ConvGRU Refines Disparity Iteratively

The filtered cost volume supplies an initial disparity, but boundaries and difficult regions still need correction. A convolutional gated recurrent unit, or ConvGRU, repeatedly examines the current disparity, matching evidence, and hidden state. Each iteration predicts an update rather than restarting the map.

If dkd_k denotes disparity after iteration kk, the process can be summarized as

dk=dk−1+Δdk.d_k=d_{k-1}+\Delta d_k.

dk−1d_{k-1} is the previous estimate and Δdk\Delta d_k is the correction predicted at iteration kk. The Telekinesis valid_iters parameter controls how many of these refinement steps run.

Recurrent networks contain linked channel dependencies: pruning one output can invalidate a later recurrent input. The paper constructs a dependency graph, ranks parameters using a first-order Taylor approximation, prunes unimportant structure consistently, and retrains the reduced refinement module.

Its retraining loss is

L=∑k=1KγK−k∥dk−dˉ∥1+λ∑i=1L∥xi−xˉi∥22.\mathcal L=\sum_{k=1}^{K}\gamma^{K-k} \lVert d_k-\bar d\rVert_1 +\lambda\sum_{i=1}^{L}\lVert x_i-\bar x_i\rVert_2^2.

KK is the number of refinement iterations; dkd_k is predicted disparity after iteration kk; dˉ\bar d is ground-truth or trusted pseudo-label disparity; γ=0.9\gamma=0.9 gives later iterations greater weight; LL is the number of refinement layers; xix_i and xˉi\bar x_i are student and teacher features at layer ii; and λ=0.1\lambda=0.1 balances output accuracy against feature distillation.

This equation teaches the two goals clearly. The first term makes every recurrent estimate geometrically accurate. The second keeps the pruned network’s internal computation near the stronger teacher.

1.7 What the Model Was Trained On

Fast-FoundationStereo uses the mixed stereo training data inherited from FoundationStereo and adds 1.4 million pseudo-labeled in-the-wild stereo pairs. FoundationStereo’s own dataset contains roughly one million high-fidelity synthetic pairs designed to include difficult materials, low-texture regions, reflections, and occlusion.

For the additional real imagery, the authors do not blindly trust one teacher prediction. Their pipeline compares stereo estimates with monocular depth through surface-normal consistency, corrects scale where appropriate, and masks unreliable sky regions. Temporal subsampling removes near-duplicate video frames. The surviving pseudo-labels add real-world visual variety while feature distillation transfers the teacher’s internal priors.

“Zero-shot” therefore means the released model is intended to work on new stereo scenes without fine-tuning for each deployment domain. It does not mean the model was trained without data, nor that every new camera, material, or weather condition will work equally well.

1.8 Where Stereo Depth Fails

Occlusion: a point visible to one camera may be hidden from the other, so no valid correspondence exists.

Weak texture: a blank surface provides few distinctive features. Context can help, but the match remains less observable.

Repetition: identical slats, shelves, or holes support several plausible disparities.

Reflections and transparency: the two cameras may observe different reflected content rather than one stable surface point.

Calibration or synchronization error: rectification assumes a rigid, known stereo pair and near-simultaneous capture. Moving objects can violate row alignment when exposures occur at different times.

Disparity range: very close objects may exceed max_disp; very distant objects may have too little disparity for reliable metric depth.

The model can generalize visual matching better than classical block matching, but it does not remove the geometry’s observability limits.

1.9 Engineering Trade-offs for a Robot

Increasing max_disp expands the near working range but grows the cost volume. Increasing valid_iters adds recurrent computation, but this distilled model is tuned around the default eight iterations; more iterations are not guaranteed to improve it. A wider physical baseline increases disparity and improves depth resolution at distance, yet also creates larger occluded regions and makes matching close objects harder. A longer focal length similarly increases disparity but narrows field of view.

Benchmark the complete sensing chain. Record capture synchronization, rectification, service latency, invalid-depth rate, and depth error by distance and material. For manipulation, report error in the final 3D point or pose rather than judging only whether the depth visualization looks smooth.

2. How to Use Fast-FoundationStereo in the Telekinesis Agentic OS

Telekinesis provides Fast-FoundationStereo as the Vitreous Skill estimate_depth_using_fast_foundation_stereo. The code below creates a simple rectified stereo pair with 20 pixels of disparity, supplies the left camera intrinsics and a 5 cm baseline, and requests metric depth.

import numpy as np

from telekinesis import datatypes, vitreous

height, width = 480, 640
disparity_px = 20
baseline = 0.05

rng = np.random.default_rng(0)
texture = rng.integers(
    0, 255, (height, width + disparity_px, 3), dtype=np.uint8
)
left_image = datatypes.Image(texture[:, :width])
right_image = datatypes.Image(
    texture[:, disparity_px : disparity_px + width]
)

camera_calibration = datatypes.CameraCalibration(
    width=width,
    height=height,
    distortion_model="plumb_bob",
    distortion_parameters=[0.0, 0.0, 0.0, 0.0, 0.0],
    intrinsic_matrix=[
        600.0, 0.0, 320.0,
        0.0, 600.0, 240.0,
        0.0, 0.0, 1.0,
    ],
)

depth_image = vitreous.estimate_depth_using_fast_foundation_stereo(
    left_image=left_image,
    right_image=right_image,
    camera_calibration=camera_calibration,
    baseline=baseline,
    valid_iters=8,
    max_disp=192,
)

valid_depth = depth_image.depth[depth_image.depth > 0]
print("Depth shape:", depth_image.depth.shape)
print("Median valid depth:", np.median(valid_depth))

The synthetic shift corresponds to 600×0.05/20=1.5600\times0.05/20=1.5 metres. Real inputs must already be rectified, share the same resolution, and come from a calibrated, synchronized stereo rig. The returned DepthImage uses metres; zero marks pixels without valid depth, and its color plane is aligned to the left image.

Try it out

Estimate stereo depth with Vitreous

See the complete Skill signature, calibration requirements, tuning controls, and return schema.

Open Fast-FoundationStereo docs

Runnable example

Run the Fast-FoundationStereo example

Open the exact Python example for producing metric depth from a rectified stereo pair.

View Python example

3. Benchmarking

3.1 Zero-Shot Results Reported in the Paper

The Fast-FoundationStereo paper evaluates all methods without fine-tuning on the target datasets. The full paper table reports bad-pixel rates at several thresholds; this compact version uses BP-2 for Middlebury, BP-1 for ETH3D, and D1 for KITTI. Lower is better. Runtime is measured on Middlebury-Q with an NVIDIA RTX 3090.

MethodMiddlebury-H BP-2 ↓Middlebury-Q BP-2 ↓ETH3D BP-1 ↓KITTI 2012 D1 ↓KITTI 2015 D1 ↓Runtime ↓
StereoAnywhere4.753.251.432.813.52427 ms
MonSter4.243.190.992.843.41336 ms
FoundationStereo1.101.300.502.302.80496 ms
RT-IGEV, same training mixture7.825.595.053.254.0045 ms
LightStereo-L, same training mixture12.557.7016.343.734.5130 ms
Fast-FoundationStereo2.202.121.222.353.2549 ms / 21 ms TensorRT

The comparison shows the intended compromise: FoundationStereo remains more accurate, but Fast-FoundationStereo cuts the paper’s RTX 3090 runtime by roughly tenfold while staying much closer to it than other real-time models.

3.2 Non-Lambertian Surfaces

Booster-Q includes reflective and transparent surfaces that violate the simple stereo assumption that corresponding pixels look alike.

MethodBP-2 ↓BP-4 ↓BP-6 ↓BP-8 ↓EPE ↓Real-time
StereoAnywhere9.015.404.123.341.21No
FoundationStereo5.184.072.912.591.13No
RT-IGEV, same training mixture18.1913.3911.3710.164.20Yes
Fast-FoundationStereo6.614.623.913.491.54Yes

3.3 Efficiency and Ablations

MethodRTX 3090RTX 4090A100ParametersMACs
FoundationStereo496 ms295 ms308 ms374.5M5,413.9G
Fast-FoundationStereo49 ms30 ms41 ms14.6M309.9G

The feature-distillation ablation shows why the paper uses mean-squared error (MSE):

Distillation strategyMiddlebury-H BP-2 ↓ETH3D BP-1 ↓KITTI 2012 D1 ↓KITTI 2015 D1 ↓
None2.872.112.674.32
Cosine similarity2.291.192.393.31
MSE2.201.222.353.25

The 1.4-million-pair pseudo-labeled dataset also improves the final model from 2.53 to 2.20 BP-2 on Middlebury-H, 1.31 to 1.22 BP-1 on ETH3D, 2.44 to 2.35 D1 on KITTI 2012, and 3.48 to 3.25 D1 on KITTI 2015.

3.4 What to Measure on a Robot

The standard disparity metric is end-point error:

EPE=1∣Ω∣∑p∈Ω∣d(p)−dˉ(p)∣.EPE=\frac{1}{|\Omega|}\sum_{p\in\Omega}|d(p)-\bar d(p)|.

Ω\Omega is the set of evaluated pixels; d(p)d(p) is predicted disparity at pixel pp; dˉ(p)\bar d(p) is reference disparity; and ∣Ω∣|\Omega| is the number of evaluated pixels. Also report the fraction exceeding a task-relevant disparity or depth error.

For robotics, stratify results by distance, material, texture, occlusion, and motion. Measure metric depth error, invalid pixels, boundary error, latency percentiles, and the accuracy of downstream 3D positions. Paper measurements do not include your capture, rectification, network transport, or target accelerator.

4. Where to Go Next

5. References