Lesson 55: Stereo Depth and FoundationStereo¶
Lesson 21 built classical stereo matching: slide a small window along corresponding scanlines, compare intensity patches, take the disparity with lowest cost. It works well on textured surfaces and fails predictably on flat, low-texture regions — the aperture problem, where a small window genuinely cannot tell which shift is correct. This lesson builds a small learned stereo network (a differentiable cost volume plus a convolutional refinement step, the core idea behind modern deep stereo systems up through FoundationStereo, Wen et al., 2025) and shows concretely why learning beats pure local matching: a network's receptive field lets it borrow context from outside the ambiguous region, something a fixed local window structurally cannot do.
import numpy as np
import cv2
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
A stereo pair with a deliberately textureless band¶
Generate a textured "world" strip, wide enough that shifting it produces a valid left/right pair at a known integer disparity — Lesson 21's synthetic stereo setup, extended with one twist: a horizontal band is flattened to constant intensity, with no texture at all inside it.
SIZE = 32
MAX_DISP = 8
def make_stereo_pair(rng, size=SIZE, max_disp=MAX_DISP, textureless=True):
world = rng.uniform(0, 1, (size, size + max_disp)).astype(np.float32)
world = cv2.GaussianBlur(world, (3, 3), 0)
for _ in range(15): # a few sharp features, so texture is patchy rather than uniform
cx, cy = rng.integers(0, size + max_disp), rng.integers(0, size)
world[max(0, cy-1):cy+2, max(0, cx-1):cx+2] = rng.uniform(0, 1)
disparity = rng.integers(1, max_disp)
band = None
if textureless:
b0 = rng.integers(0, size - size // 3)
band = slice(b0, b0 + size // 3)
world[band, :] = 0.5 # a flat strip: no texture, so no local cue to match against
left = world[:, max_disp:max_disp + size]
right = world[:, max_disp - disparity: max_disp - disparity + size]
disp_map = np.full((size, size), float(disparity), dtype=np.float32)
return left.astype(np.float32), right.astype(np.float32), disp_map, band
rng = np.random.default_rng(11)
left0, right0, disp0, band0 = make_stereo_pair(rng)
fig, axes = plt.subplots(1, 3, figsize=(9, 3))
axes[0].imshow(left0, cmap='gray'); axes[0].set_title('left image'); axes[0].axis('off')
axes[1].imshow(right0, cmap='gray'); axes[1].set_title('right image'); axes[1].axis('off')
axes[2].imshow(disp0, cmap='viridis'); axes[2].set_title(f'true disparity ({disp0[0,0]:.0f} px)'); axes[2].axis('off')
for ax in axes:
ax.axhline(band0.start, color='red', linestyle='--', linewidth=0.7)
ax.axhline(band0.stop, color='red', linestyle='--', linewidth=0.7)
plt.show()
Classical block matching, and where it fails¶
Lesson 21's block matching: for every pixel, compare a small window against candidate windows at every disparity, keep the lowest-cost match.
def block_match(left, right, max_disp=MAX_DISP, win=3):
H, W = left.shape
half = win // 2
disp_map = np.zeros((H, W), dtype=np.float32)
left_pad = np.pad(left, half, mode='edge')
right_pad = np.pad(right, half, mode='edge')
for y in range(H):
for x in range(W):
patch_l = left_pad[y:y+win, x:x+win]
best_d, best_cost = 0, np.inf
for d in range(max_disp):
x2 = x + d
if x2 + win > right_pad.shape[1]:
continue
patch_r = right_pad[y:y+win, x2:x2+win]
cost = np.sum((patch_l - patch_r) ** 2)
if cost < best_cost:
best_cost, best_d = cost, d
disp_map[y, x] = best_d
return disp_map
bm_disp0 = block_match(left0, right0)
err_textured = np.abs(bm_disp0[:band0.start, :] - disp0[:band0.start, :]).mean()
err_textureless = np.abs(bm_disp0[band0, :] - disp0[band0, :]).mean()
print(f'block matching MAE, textured region: {err_textured:.2f} px')
print(f'block matching MAE, textureless region: {err_textureless:.2f} px (true disparity = {disp0[0,0]:.0f})')
Block matching is nearly exact where texture exists, and badly wrong inside the flat band — a small window there genuinely contains no information to disambiguate disparity, so it locks onto whatever noise happens to look marginally better. No amount of clever thresholding fixes this: the window is too small to see anything else.
A learned cost volume¶
Deep stereo networks (PSMNet, RAFT-Stereo, and now FoundationStereo) replace hand-picked window comparison with three learned pieces:
- A small CNN extracts a feature map from each image (not raw pixels — learned features, more robust than intensity alone).
- A cost volume is built by correlating the left feature map against the right feature map, shifted by every candidate disparity — the same all-disparities-at-once idea as block matching's search, just done at the feature level.
- A refinement network (a few more conv layers, working across the whole cost volume) converts a soft, smooth probability distribution over disparities into a final estimate — this step has a receptive field spanning far more of the image than any local window, so it can borrow context from well outside an ambiguous region.
class FeatureNet(nn.Module):
def __init__(self, out_ch=8):
super().__init__()
self.net = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(), nn.Conv2d(16, out_ch, 3, padding=1))
def forward(self, x):
return self.net(x)
class StereoNet(nn.Module):
def __init__(self, max_disp=MAX_DISP):
super().__init__()
self.feat = FeatureNet()
self.max_disp = max_disp
self.refine = nn.Sequential(
nn.Conv2d(max_disp, 16, 3, padding=1), nn.ReLU(),
nn.Conv2d(16, max_disp, 3, padding=1),
)
def forward(self, left, right):
fl, fr = self.feat(left), self.feat(right)
cost_volume = []
for d in range(self.max_disp):
shifted = fr if d == 0 else F.pad(fr[:, :, :, :-d], (d, 0))
cost_volume.append((fl * shifted).sum(dim=1)) # correlation at this disparity
cost_volume = self.refine(torch.stack(cost_volume, dim=1)) # regularize across space
probs = F.softmax(cost_volume, dim=1)
disp_range = torch.arange(self.max_disp, dtype=torch.float32).view(1, -1, 1, 1)
return (probs * disp_range).sum(dim=1) # soft-argmin: expected disparity, sub-pixel
rng2 = np.random.default_rng(11)
N = 300
lefts, rights, disp_maps = [], [], []
for _ in range(N):
l, r, d, _ = make_stereo_pair(rng2)
lefts.append(l); rights.append(r); disp_maps.append(d)
lefts, rights, disp_maps = np.array(lefts), np.array(rights), np.array(disp_maps)
split = int(0.85 * N)
Ltr, Rtr, Dtr = lefts[:split], rights[:split], disp_maps[:split]
Lte, Rte, Dte = lefts[split:], rights[split:], disp_maps[split:]
torch.manual_seed(0)
model = StereoNet()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
Lt = torch.tensor(Ltr).unsqueeze(1); Rt = torch.tensor(Rtr).unsqueeze(1); Dt = torch.tensor(Dtr)
for _ in range(300):
opt.zero_grad()
loss = F.l1_loss(model(Lt, Rt), Dt)
loss.backward()
opt.step()
with torch.no_grad():
pred_te = model(torch.tensor(Lte).unsqueeze(1), torch.tensor(Rte).unsqueeze(1))
mae = (pred_te - torch.tensor(Dte)).abs().mean().item()
print(f'learned stereo network MAE (overall, includes textureless bands): {mae:.3f} px')
Head to head, on the textureless band specifically¶
Compare both methods on a single fresh pair, restricted to exactly the pixels inside the flat, featureless band — precisely where block matching's local window has nothing to go on.
test_rng = np.random.default_rng(99)
left1, right1, disp1, band1 = make_stereo_pair(test_rng)
bm_disp1 = block_match(left1, right1)
with torch.no_grad():
learned_disp1 = model(torch.tensor(left1[None, None]), torch.tensor(right1[None, None]))[0].numpy()
bm_err_band = np.abs(bm_disp1[band1, :] - disp1[band1, :]).mean()
learned_err_band = np.abs(learned_disp1[band1, :] - disp1[band1, :]).mean()
print(f'textureless-band MAE: block matching = {bm_err_band:.2f} px, learned = {learned_err_band:.2f} px')
fig, axes = plt.subplots(1, 4, figsize=(11, 3))
axes[0].imshow(left1, cmap='gray'); axes[0].set_title('left'); axes[0].axis('off')
axes[1].imshow(disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[1].set_title('true disparity'); axes[1].axis('off')
axes[2].imshow(bm_disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[2].set_title('block matching'); axes[2].axis('off')
axes[3].imshow(learned_disp1, cmap='viridis', vmin=0, vmax=MAX_DISP); axes[3].set_title('learned'); axes[3].axis('off')
for ax in axes:
ax.axhline(band1.start, color='red', linestyle='--', linewidth=0.7)
ax.axhline(band1.stop, color='red', linestyle='--', linewidth=0.7)
plt.show()
The learned network's error inside the flat band is far lower than block matching's. Nothing in the band itself became less ambiguous — the refinement network's convolutions have a receptive field extending well beyond the band's own pixels, so the disparity estimate for a pixel in the middle of the flat region is influenced by evidence from the textured pixels around and above it. A fixed 3x3 window can never do this; it only ever sees what's directly inside it.
FoundationStereo, and what "foundation" adds here¶
FoundationStereo (NVIDIA, 2025) is this same recipe — feature extraction, cost volume, learned refinement (via an iterative update scheme derived from RAFT and RAFT-Stereo, rather than the one-shot softmax used here) — trained at foundation-model scale on a mix of large synthetic stereo datasets and real captured data, specifically to generalize to new cameras, scenes, and domains without per-dataset fine-tuning. That's the same "learn a prior at scale, use it zero-shot" story as Lesson 49's CLIP and Lesson 51's SAM, now applied to metric depth from a calibrated stereo pair rather than a single monocular image — which sidesteps Lesson 53's fundamental scale-ambiguity problem entirely, since a calibrated stereo baseline gives disparity a real geometric meaning (Lesson 21-23) that a single image never has.
Exercise¶
- Increase the textureless band's width from
size // 3tosize // 2inmake_stereo_pair. Does the learned network's advantage over block matching grow, shrink, or stay about the same as the ambiguous region gets larger relative to the image? StereoNet'srefinestep is two convolutional layers with a 3x3 kernel each, giving a limited receptive field. Add a third conv layer and rerun the textureless-band comparison — does a larger receptive field (more surrounding context reachable) improve the band's disparity estimate further?- Remove the
refinenetwork entirely (useprobs = F.softmax(cost_volume, dim=1)directly on the raw correlation cost volume, with no learned regularization step) and retrain. Does the network's textureless-band performance collapse back toward block matching's, confirming that spatial context — not the learned features alone — is what closes the gap?