Lesson 54: Depth Anything¶
Lesson 53 showed monocular depth's central weakness: a network trained on a small, narrow dataset only learns priors valid within that dataset's distribution, and gets confidently fooled outside it. Depth Anything (Yang et al., 2024) is a monocular depth foundation model built specifically to close that gap, using self-training on unlabeled data: a teacher model pseudo-labels a huge pool of unlabeled images, and a student trains on labeled data plus those pseudo-labels, at a scale (over 60 million unlabeled images) no manually-annotated depth dataset gets close to. This lesson builds that pipeline — and its result is a genuine, useful negative finding: pseudo-labeling from a biased teacher does not, by itself, fix the bias. Understanding why is what actually explains what Depth Anything gets right that a naive version of the same idea doesn't.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
Setup: a small labeled set, a large unlabeled pool, a distribution gap¶
Reuse Lesson 53's size-cue depth scenes. The labeled set is small (20 scenes) and covers only a narrow depth range (0.4-0.6). The unlabeled pool is much larger (150 scenes, no depth labels — just images) and, like real-world unlabeled photo collections, spans the full range of conditions (depths 0.2-0.9). The test set matches that full range too — deliberately including depths the labeled set never showed.
SIZE = 32
def make_scene(rng, size=SIZE, n_objects=4, depth_range=(0.2, 0.9)):
img = np.zeros((size, size), dtype=np.float32)
depth = np.full((size, size), 1.0, dtype=np.float32)
depths_used = rng.uniform(*depth_range, n_objects)
depths_used.sort()
for d in depths_used[::-1]:
r = max(2, int(6 * (1 - d) + 1))
cy = int(size * (0.3 + 0.6 * d))
cx = rng.integers(r, size - r)
yy, xx = np.mgrid[0:size, 0:size]
m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2
img[m] = 0.8
depth[m] = d
img = np.clip(img + rng.normal(0, 0.03, img.shape), 0, 1).astype(np.float32)
return img, depth
label_rng = np.random.default_rng(10)
N_LABELED = 20
X_labeled, D_labeled = [], []
for _ in range(N_LABELED):
im, d = make_scene(label_rng, n_objects=label_rng.integers(2, 4), depth_range=(0.4, 0.6))
X_labeled.append(im); D_labeled.append(d)
X_labeled, D_labeled = np.array(X_labeled, dtype=np.float32), np.array(D_labeled, dtype=np.float32)
unlabeled_rng = np.random.default_rng(20)
N_UNLABELED = 150
X_unlabeled = []
for _ in range(N_UNLABELED):
im, _ = make_scene(unlabeled_rng, n_objects=unlabeled_rng.integers(2, 6), depth_range=(0.2, 0.9))
X_unlabeled.append(im)
X_unlabeled = np.array(X_unlabeled, dtype=np.float32)
test_rng = np.random.default_rng(30)
N_TEST = 100
X_test, D_test = [], []
for _ in range(N_TEST):
im, d = make_scene(test_rng, n_objects=test_rng.integers(2, 6), depth_range=(0.2, 0.9))
X_test.append(im); D_test.append(d)
X_test, D_test = np.array(X_test, dtype=np.float32), np.array(D_test, dtype=np.float32)
print(f'labeled: {N_LABELED} scenes, depth range [0.4, 0.6]')
print(f'unlabeled: {N_UNLABELED} scenes, depth range [0.2, 0.9], no depth labels')
print(f'test: {N_TEST} scenes, depth range [0.2, 0.9]')
Teacher, pseudo-labels, student¶
Train a teacher (Lesson 41's U-Net) on the small labeled set only. Use it to pseudo-label every image in the unlabeled pool. Train a student, identical architecture, on the labeled set plus the pseudo-labeled pool combined — exactly Depth Anything's core recipe.
class UNetTiny(nn.Module):
def __init__(self):
super().__init__()
self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())
self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())
self.pool = nn.MaxPool2d(2)
self.up = nn.Upsample(scale_factor=2, mode='nearest')
self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())
self.out = nn.Conv2d(16, 1, 1)
def forward(self, x):
f1 = self.enc1(x)
f2 = self.enc2(self.pool(f1))
d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))
return torch.sigmoid(self.out(d1)).squeeze(1)
def train_model(X, D, seed, epochs=200, lr=0.01):
torch.manual_seed(seed)
model = UNetTiny()
opt = torch.optim.Adam(model.parameters(), lr=lr)
Xt, Dt = torch.tensor(X).unsqueeze(1), torch.tensor(D)
for _ in range(epochs):
opt.zero_grad()
loss = F.mse_loss(model(Xt), Dt)
loss.backward()
opt.step()
return model
def eval_mae(model, X, D):
with torch.no_grad():
pred = model(torch.tensor(X).unsqueeze(1))
return (pred - torch.tensor(D)).abs().mean().item()
teacher = train_model(X_labeled, D_labeled, seed=0)
teacher_mae = eval_mae(teacher, X_test, D_test)
print(f'teacher (small labeled set only) test MAE: {teacher_mae:.4f}')
with torch.no_grad():
pseudo_labels = teacher(torch.tensor(X_unlabeled).unsqueeze(1)).numpy()
X_combined = np.concatenate([X_labeled, X_unlabeled])
D_combined = np.concatenate([D_labeled, pseudo_labels])
student = train_model(X_combined, D_combined, seed=1)
student_mae = eval_mae(student, X_test, D_test)
print(f'student (labeled + pseudo-labeled) test MAE: {student_mae:.4f}')
Why didn't self-training help?¶
The student's error is essentially unchanged from the teacher's. This is not a bug — it's the expected outcome of naive self-training, and it's worth understanding precisely why. The pseudo-labels for the unlabeled pool came entirely from the teacher, and the teacher was never shown a depth outside [0.4, 0.6]. For any unlabeled scene with a true depth of, say, 0.8, the teacher's pseudo-label isn't a noisy version of the right answer — it's the teacher's same systematic extrapolation error from Lesson 53's fooling experiment, confidently applied and then handed to the student as if it were ground truth. Training on that just teaches the student to reproduce the teacher's mistake more efficiently. No genuinely new information about depths in [0.6, 0.9] ever entered the pipeline — self-training redistributes what a model already knows, and cannot manufacture what it doesn't.
What actually makes Depth Anything work¶
The gap between this toy failure and Depth Anything's real, well-documented success comes from three ingredients this lesson's setup deliberately lacks:
- Genuine distributional coverage. Depth Anything's unlabeled pool is not a narrow synthetic distribution shifted slightly from the labeled set — it's 62 million real images spanning an enormous range of scenes, lighting, and object types. The teacher's errors on any single unlabeled image are still real errors, but averaged over that much genuine diversity, the student's training signal is far less systematically biased than this lesson's 150-image, single-cue toy pool.
- A strong pretrained backbone. The teacher isn't a small CNN trained from scratch on a handful of labeled examples — it starts from a DINOv2 (Lesson 48) backbone already carrying rich, general-purpose visual priors learned from unlabeled data at a completely different scale, before ever seeing a single depth label.
- Aggressive perturbation during student training. Depth Anything specifically injects strong image augmentations (and, in later variants, additional auxiliary losses) when training the student on pseudo-labeled data, so the student cannot simply memorize the teacher's exact outputs — it has to learn something more robust to reproduce them under distortion, which is what actually squeezes new generalization out of the process, rather than merely copying the teacher.
None of these fix this toy pipeline's fundamental problem (there is no genuinely new information about depths in [0.6, 0.9] anywhere in this dataset), but at real scale, "genuinely new information" is rarely completely absent from a sufficiently large and diverse unlabeled pool — which is exactly the condition self-training needs to be worth doing.
Exercise¶
- Change the unlabeled pool's
depth_rangeto match the labeled set's(0.4, 0.6)exactly, instead of the full(0.2, 0.9)range. Does self-training help now — and does that support the claim that self-training's value depends entirely on whether the unlabeled pool's true (unobserved) labels actually differ from what the teacher already believes? - Add Gaussian pixel noise to the unlabeled images before generating pseudo-labels but train the student on the clean images with those noisy-derived labels (or vice versa) — a crude stand-in for Depth Anything's perturbation trick. Does forcing this mismatch between what's pseudo-labeled and what's trained on change the student's test MAE at all?
- Retrain the teacher on a labeled set covering the full
(0.2, 0.9)depth range (same 20 examples, just resample withdepth_range=(0.2, 0.9)) instead of the narrow one. Does self-training help this teacher improve further on the same test set, now that there's no fundamental information gap to begin with?