Lesson 53: Monocular Depth Estimation¶

Lesson 21's stereo matching recovered depth from two images by measuring disparity — a direct geometric signal. Monocular depth estimation asks for the same per-pixel depth map from a single image, with no disparity available at all. This is fundamentally ill-posed: infinitely many 3D scenes produce the exact same 2D image (a toy car photographed up close is indistinguishable, pixel-for-pixel, from a real car photographed from far away). Modern monocular depth networks work anyway, by learning statistical priors about the world — typical object sizes, perspective, occlusion — from massive training sets. This lesson builds a small depth network from one such prior (size/perspective cues), and then deliberately breaks the prior to show exactly where the ill-posedness the intro paragraph mentioned actually bites.

In [1]:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt

A scene with a size-based depth cue¶

Circles at different depths, with deliberately uniform brightness — the only information available about depth is each circle's size (closer objects are drawn larger) and its vertical position (closer objects sit lower in the frame, a simple ground-plane perspective cue). This is a synthetic stand-in for the single most important prior a real monocular depth network learns: how large an object of a given kind "should" look at a given distance.

In [2]:

SIZE = 32

def make_scene(rng, size=SIZE, n_objects=4):
    img = np.zeros((size, size), dtype=np.float32)
    depth = np.full((size, size), 1.0, dtype=np.float32)  # background = far (depth = 1)
    depths_used = rng.uniform(0.2, 0.9, n_objects)
    depths_used.sort()
    for d in depths_used[::-1]:  # draw closest (smallest depth) last, so it occludes farther ones
        r = max(2, int(6 * (1 - d) + 1))       # closer (small d) -> larger radius
        cy = int(size * (0.3 + 0.6 * d))
        cx = rng.integers(r, size - r)
        yy, xx = np.mgrid[0:size, 0:size]
        m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2
        img[m] = 0.8  # uniform intensity: size/position are the ONLY depth cues
        depth[m] = d
    img = np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1).astype(np.float32)
    return img, depth

rng = np.random.default_rng(3)
N = 300
imgs, depths = [], []
for _ in range(N):
    im, d = make_scene(rng)
    imgs.append(im); depths.append(d)
imgs = np.array(imgs, dtype=np.float32)
depths = np.array(depths, dtype=np.float32)

split = int(0.85 * N)
Xtr, Dtr = imgs[:split], depths[:split]
Xte, Dte = imgs[split:], depths[split:]

fig, axes = plt.subplots(2, 4, figsize=(9, 4.5))
for i in range(4):
    axes[0, i].imshow(Xtr[i], cmap='gray'); axes[0, i].axis('off')
    axes[1, i].imshow(Dtr[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[1, i].axis('off')
axes[0, 0].set_title('image', fontsize=9, loc='left')
axes[1, 0].set_title('true depth (bright=near)', fontsize=9, loc='left')
plt.show()

No description has been provided for this image

A depth-regression U-Net¶

Depth estimation is per-pixel regression rather than per-pixel classification (Lesson 41) — the exact same encoder-decoder-with-skip-connections architecture, with a single continuous output per pixel instead of a class distribution.

In [3]:

class UNetTiny(nn.Module):
    def __init__(self):
        super().__init__()
        self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())
        self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())
        self.pool = nn.MaxPool2d(2)
        self.up = nn.Upsample(scale_factor=2, mode='nearest')
        self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())
        self.out = nn.Conv2d(16, 1, 1)

    def forward(self, x):
        f1 = self.enc1(x)
        f2 = self.enc2(self.pool(f1))
        d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))
        return torch.sigmoid(self.out(d1)).squeeze(1)  # depth in (0, 1)

torch.manual_seed(0)
model = UNetTiny()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
Xt = torch.tensor(Xtr).unsqueeze(1); Dt = torch.tensor(Dtr)
for _ in range(300):
    opt.zero_grad()
    loss = F.mse_loss(model(Xt), Dt)
    loss.backward()
    opt.step()

with torch.no_grad():
    pred_te = model(torch.tensor(Xte).unsqueeze(1))

mae = (pred_te - torch.tensor(Dte)).abs().mean().item()
mean_depth_baseline = np.abs(Dte - Dtr.mean()).mean()
print(f'mean absolute depth error:                        {mae:.4f}  (depth range is [0.2, 1.0])')
print(f'baseline (predict the mean training depth everywhere): {mean_depth_baseline:.4f}')

mean absolute depth error:                        0.0081  (depth range is [0.2, 1.0])
baseline (predict the mean training depth everywhere): 0.1233

In [4]:

fig, axes = plt.subplots(3, 4, figsize=(9, 6.5))
for i in range(4):
    axes[0, i].imshow(Xte[i], cmap='gray'); axes[0, i].axis('off')
    axes[1, i].imshow(Dte[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[1, i].axis('off')
    axes[2, i].imshow(pred_te[i], cmap='viridis_r', vmin=0.2, vmax=1.0); axes[2, i].axis('off')
for r, name in enumerate(['input', 'true depth', 'predicted depth']):
    axes[r, 0].set_title(name, fontsize=9, loc='left')
plt.tight_layout()
plt.show()

Breaking the prior: the "toy car" problem¶

The network has clearly learned "bigger circle = closer" — but that's a learned statistical association, not a measurement. If an object's size doesn't follow the training distribution's rule (a miniature scale model, a projected image, a genuinely huge object) the network has no way to tell, because it never had access to true 3D geometry in the first place. Test this directly: two circles with the identical true depth, but one sized correctly for that depth and one sized as if it were much closer.

In [5]:

def make_single_object(cx, cy, r, size=SIZE):
    img = np.zeros((size, size), dtype=np.float32)
    yy, xx = np.mgrid[0:size, 0:size]
    mask = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2
    img[mask] = 0.8
    return img, mask

true_depth = 0.8  # actually far away, in both cases below
normal_r = max(2, int(6 * (1 - true_depth) + 1))   # the radius a far object should have
faked_r = max(2, int(6 * (1 - 0.2) + 1))            # the radius as if it were very close instead

img_normal, mask_normal = make_single_object(16, 16, normal_r)
img_faked, mask_faked = make_single_object(16, 16, faked_r)

with torch.no_grad():
    pred_normal = model(torch.tensor(img_normal[None, None]))[0].numpy()
    pred_faked = model(torch.tensor(img_faked[None, None]))[0].numpy()

print(f'true depth in both cases: {true_depth}  (identical — nothing about the actual geometry changed)')
print(f'predicted depth, correctly-sized-for-its-depth object (r={normal_r}): {pred_normal[mask_normal].mean():.3f}')
print(f'predicted depth, oversized object (r={faked_r}, same true depth):     {pred_faked[mask_faked].mean():.3f}')

fig, axes = plt.subplots(1, 2, figsize=(6, 3))
axes[0].imshow(img_normal, cmap='gray'); axes[0].set_title(f'normal size\npred depth={pred_normal[mask_normal].mean():.2f}', fontsize=9)
axes[1].imshow(img_faked, cmap='gray'); axes[1].set_title(f'oversized\npred depth={pred_faked[mask_faked].mean():.2f}', fontsize=9)
for ax in axes: ax.axis('off')
plt.show()

true depth in both cases: 0.8  (identical — nothing about the actual geometry changed)
predicted depth, correctly-sized-for-its-depth object (r=2): 0.813
predicted depth, oversized object (r=5, same true depth):     0.331

The network confidently reports two very different depths for two objects at the exact same true distance — it isn't measuring depth, it's pattern-matching against "how big things of this kind usually look at this distance," and that pattern-match can be fooled by anything that violates the training distribution's assumptions (miniatures, forced-perspective photography, an object of an unusual size). This is the real, well-documented failure mode of monocular depth networks in practice, and it is exactly what "ill-posed without additional information" predicted at the start of this lesson: the 2D image alone never contained the true depth, only cues correlated with it in typical training data.

The practical fix real systems use is exactly the one this lesson opened by contrasting against: whenever a second geometric measurement is available — a second camera (Lesson 21's stereo, Lesson 55), LiDAR, or known camera motion (structure from motion, Lesson 28) — it should be trusted over a monocular size prior. Monocular depth is most valuable precisely where those aren't available (a single photograph, a single video frame with no motion baseline), and least trustworthy exactly where its training-distribution assumptions break down.

Exercise¶

Retrain with brightness restored as a depth cue (img[m] = 0.4 + 0.5 * (1 - d), matching the very first version of this experiment) instead of uniform intensity. Rerun the fooling test — does the size-based illusion get weaker, because the model now has a second, un-fooled cue (brightness) to fall back on?
The silog (scale-invariant log RMSE) metric from real monocular-depth research measures error up to a global multiplicative scale factor, since monocular predictions are often only correct up to scale. Implement it (sqrt(mean((log(pred) - log(true))^2) - mean(log(pred) - log(true))^2)) and compare it to plain MAE — does the model's ranking of "how good is this prediction" change between the two metrics?
Train a second model on scenes with n_objects fixed to exactly 1 instead of drawn from make_scene's default. Does the single-object model's depth-from-size prior transfer to the original multi-object test scenes, or does it fail specifically when circles occlude each other?

← Lesson 52: Open-Vocabulary Detection

Lesson 54: Depth Anything →