Lesson 56: Feed-Forward Camera Pose Estimation¶
Lesson 22's RANSAC and Lesson 28's structure from motion recover camera geometry through an iterative pipeline: detect features, match them, run RANSAC to reject outliers, solve for the essential matrix, decompose it into rotation and translation. Each stage is a separate, hand-designed algorithm. VGGT (Visual Geometry Grounded Transformer, Wang et al., 2025) replaces the entire pipeline with a single feed-forward network: feed it correspondences (or even raw images) from multiple views, and it directly regresses camera poses in one forward pass, with no explicit RANSAC step anywhere. This lesson builds a small version of that idea and compares it against the classical pipeline it replaces — including exactly where each one wins.
import numpy as np
import cv2
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
Synthetic two-view correspondences, with outliers¶
A random 3D point cloud, projected into two camera views related by a known random rotation and translation direction (Lesson 25's pinhole projection). Translation direction is the most a calibrated two-view pair can ever recover — absolute scale is fundamentally unobservable from two views alone (the same monocular scale ambiguity from Lesson 53, in a different guise). A controllable fraction of the correspondences are replaced with random mismatches, simulating the false matches Lesson 22's RANSAC was built to survive.
K = np.array([[50., 0, 32], [0, 50., 32], [0, 0, 1]])
N_POINTS = 40
def random_rotation(rng, max_angle_deg=30):
axis = rng.normal(size=3); axis /= np.linalg.norm(axis)
angle = np.radians(rng.uniform(-max_angle_deg, max_angle_deg))
Kx = np.array([[0, -axis[2], axis[1]], [axis[2], 0, -axis[0]], [-axis[1], axis[0], 0]])
return np.eye(3) + np.sin(angle) * Kx + (1 - np.cos(angle)) * (Kx @ Kx)
def make_pair(rng, n_points=N_POINTS, outlier_frac=0.0):
pts3d = rng.uniform(-2, 2, (n_points, 3)) + np.array([0, 0, 8])
R_true = random_rotation(rng)
t_dir = rng.uniform(-1, 1, 3); t_dir /= np.linalg.norm(t_dir)
def project(pts, R, t):
cam_pts = (R @ pts.T).T + t
proj = (K @ cam_pts.T).T
return proj[:, :2] / proj[:, 2:3]
pts1 = project(pts3d, np.eye(3), np.zeros(3))
pts2 = project(pts3d, R_true, t_dir * 2.0)
n_outliers = int(outlier_frac * n_points)
if n_outliers > 0:
idx = rng.choice(n_points, n_outliers, replace=False)
pts2[idx] = rng.uniform(0, 64, (n_outliers, 2))
K_inv = np.linalg.inv(K) # normalize to camera coordinates, so the network never needs to learn K
npts1 = (K_inv @ np.hstack([pts1, np.ones((n_points, 1))]).T).T[:, :2]
npts2 = (K_inv @ np.hstack([pts2, np.ones((n_points, 1))]).T).T[:, :2]
return npts1.astype(np.float32), npts2.astype(np.float32), R_true.astype(np.float32), t_dir.astype(np.float32)
def rotation_angle_error(R_est, R_true):
R_diff = R_est.T @ R_true
cos_angle = np.clip((np.trace(R_diff) - 1) / 2, -1, 1)
return np.degrees(np.arccos(cos_angle))
def translation_angle_error(t_est, t_true):
t_est = t_est / (np.linalg.norm(t_est) + 1e-8)
t_true = t_true / np.linalg.norm(t_true)
return np.degrees(np.arccos(np.clip(np.dot(t_est, t_true), -1, 1)))
The classical pipeline: essential matrix + RANSAC¶
cv2.findEssentialMat with RANSAC (Lesson 22) followed by cv2.recoverPose (Lesson 26) — the standard textbook approach to two-view relative pose, and the exact pipeline VGGT-style networks are built to replace.
def classical_pose(npts1, npts2):
pts1_px = (K @ np.hstack([npts1, np.ones((N_POINTS, 1))]).T).T[:, :2].astype(np.float32)
pts2_px = (K @ np.hstack([npts2, np.ones((N_POINTS, 1))]).T).T[:, :2].astype(np.float32)
E, mask = cv2.findEssentialMat(pts1_px, pts2_px, K, method=cv2.RANSAC, prob=0.999, threshold=1.0)
if E is None:
return np.eye(3), np.array([0, 0, 1.0])
_, R, t, _ = cv2.recoverPose(E, pts1_px, pts2_px, K)
return R, t.ravel()
rng = np.random.default_rng(0)
for outlier_frac in [0.0, 0.2, 0.4]:
rot_errs, trans_errs = [], []
for _ in range(20):
p1, p2, R_true, t_true = make_pair(rng, outlier_frac=outlier_frac)
R_est, t_est = classical_pose(p1, p2)
rot_errs.append(rotation_angle_error(R_est, R_true))
trans_errs.append(translation_angle_error(t_est, t_true))
print(f'outlier_frac={outlier_frac}: classical rot err={np.mean(rot_errs):.2f} deg, '
f'trans dir err={np.mean(trans_errs):.2f} deg')
A feed-forward pose network¶
Every correspondence (x1, y1, x2, y2) becomes a token; a small Transformer encoder (Lesson 45) lets every correspondence attend to every other one — this is the piece a naive PointNet-style max-pooling architecture is missing: recovering a two-view geometric relationship (especially translation direction) fundamentally requires reasoning about how correspondences relate to each other, not just processing each one independently and pooling. Rotation is regressed via a continuous 6D representation (Zhou et al., 2019 — two 3D vectors, Gram-Schmidt-orthonormalized into a valid rotation matrix, avoiding the discontinuities of quaternions or Euler angles), and translation direction as a unit 3-vector.
def rot6d_to_matrix(x):
a1, a2 = x[..., :3], x[..., 3:]
b1 = F.normalize(a1, dim=-1)
b2 = a2 - (b1 * a2).sum(-1, keepdim=True) * b1
b2 = F.normalize(b2, dim=-1)
b3 = torch.cross(b1, b2, dim=-1)
return torch.stack([b1, b2, b3], dim=-1)
class PoseNet(nn.Module):
def __init__(self, hidden=64, n_heads=4, n_layers=2):
super().__init__()
self.embed = nn.Linear(4, hidden)
layer = nn.TransformerEncoderLayer(hidden, n_heads, dim_feedforward=hidden * 2,
batch_first=True, dropout=0.0)
self.encoder = nn.TransformerEncoder(layer, num_layers=n_layers)
self.head = nn.Sequential(nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 9))
def forward(self, corr): # corr: (B, N, 4) = [x1, y1, x2, y2]
tok = self.encoder(self.embed(corr))
pooled = tok.mean(dim=1)
out = self.head(pooled)
R = rot6d_to_matrix(out[..., :6])
t = F.normalize(out[..., 6:], dim=-1)
return R, t
def make_batch(rng, batch_size, outlier_frac):
corrs, Rs, ts = [], [], []
for _ in range(batch_size):
p1, p2, R, t = make_pair(rng, outlier_frac=outlier_frac)
corrs.append(np.concatenate([p1, p2], axis=1))
Rs.append(R); ts.append(t)
return (torch.tensor(np.array(corrs)), torch.tensor(np.array(Rs)), torch.tensor(np.array(ts)))
torch.manual_seed(0)
model = PoseNet()
opt = torch.optim.Adam(model.parameters(), lr=0.001)
train_rng = np.random.default_rng(500)
for epoch in range(1400):
outlier_frac = train_rng.uniform(0, 0.5) # train across a range of outlier ratios
corr, R_true, t_true = make_batch(train_rng, 32, outlier_frac)
R_pred, t_pred = model(corr)
rot_loss = ((R_pred - R_true) ** 2).sum(dim=(1, 2)).mean()
trans_loss = (1 - (t_pred * t_true).sum(-1)).mean()
loss = rot_loss + trans_loss
opt.zero_grad()
loss.backward()
opt.step()
print(f'final training loss: {loss.item():.4f}')
Head to head: classical RANSAC vs. feed-forward network, across outlier ratios¶
eval_rng = np.random.default_rng(999)
results = {}
for outlier_frac in [0.0, 0.2, 0.4]:
learned_rot, learned_trans, classical_rot, classical_trans = [], [], [], []
for _ in range(20):
p1, p2, R_true, t_true = make_pair(eval_rng, outlier_frac=outlier_frac)
with torch.no_grad():
corr = torch.tensor(np.concatenate([p1, p2], axis=1))[None]
R_pred, t_pred = model(corr)
learned_rot.append(rotation_angle_error(R_pred[0].numpy(), R_true))
learned_trans.append(translation_angle_error(t_pred[0].numpy(), t_true))
R_c, t_c = classical_pose(p1, p2)
classical_rot.append(rotation_angle_error(R_c, R_true))
classical_trans.append(translation_angle_error(t_c, t_true))
results[outlier_frac] = (np.mean(learned_rot), np.mean(learned_trans), np.mean(classical_rot), np.mean(classical_trans))
print(f'outlier_frac={outlier_frac}:')
print(f' learned: rot err={np.mean(learned_rot):.2f} deg, trans dir err={np.mean(learned_trans):.2f} deg')
print(f' classical: rot err={np.mean(classical_rot):.2f} deg, trans dir err={np.mean(classical_trans):.2f} deg')
Where each approach wins¶
On clean data (no outliers), classical RANSAC wins clearly — an exact, well-conditioned geometric solve is hard to beat when the input genuinely satisfies its assumptions. As the outlier fraction increases, the gap closes and then reverses: the feed-forward network, trained across a whole range of outlier ratios, becomes more robust than RANSAC with a fixed threshold and iteration budget. This isn't a fluke of this toy setup — it's the actual, documented motivation for VGGT-style architectures: RANSAC's robustness is bounded by its hyperparameters and by the geometric model it assumes (a single rigid essential matrix relating exactly two views), while a network trained on enough varied, messy, real-world correspondence data learns something closer to "what does a plausible pose look like given evidence like this," which degrades more gracefully.
This mirrors Lesson 55's stereo story almost exactly: a fixed, hand-designed algorithm (block matching, RANSAC) is precise under its ideal assumptions and brittle outside them; a network trained across a distribution of conditions trades a little of that peak precision for much better robustness across the full range of real conditions it will actually see. Neither replaces the other outright — production 3D vision pipelines increasingly use exactly this kind of feed-forward network as a fast, robust initialization, with a classical geometric refinement (bundle adjustment, Lesson 28) as a final, exact-precision cleanup step where compute allows.
Exercise¶
- Increase
max_angle_deginrandom_rotationfrom 30 to 90. Does the gap between learned and classical performance change, and does classical RANSAC's known preference for small, well-conditioned baselines explain the direction of the shift? - Remove the self-attention (
self.encoder(self.embed(corr))→ justself.embed(corr), then mean-pool directly) and retrain. Does removing attention between correspondences hurt rotation accuracy, translation accuracy, or both — and does that match the claim that translation direction specifically needs relational reasoning across points? - Train two separate models, one only ever seeing
outlier_frac=0.0during training and one only ever seeingoutlier_frac=0.5, then evaluate both across the full[0.0, 0.5]range. Does either specialist beat the general-purpose model (trained across the whole range) on its own home turf, and does the generalist lose much by comparison anywhere?