Lesson 58: Neural Rendering¶
Lesson 28's structure from motion recovers a sparse 3D point cloud from multiple images. Neural rendering — most famously NeRF (Mildenhall et al., 2020) — asks for something richer: a continuous representation of an entire 3D scene, dense enough to render a photorealistic image from any camera viewpoint, including ones never observed during training. The representation isn't a mesh or a point cloud at all; it's the weights of a small neural network. This lesson builds one from scratch: a coordinate network trained purely by comparing rendered pixels to real ones, with no 3D supervision anywhere in the loss.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
Ground truth: a classical ray-tracer for a synthetic scene¶
A single shaded sphere, photographed from 8 camera positions arranged in a ring around it (Lesson 25's pinhole camera model and image formation). Each pixel's color comes from a classical, non-learned ray-sphere intersection with simple directional shading — this is only used to generate training data and a ground-truth held-out view to grade against; the network itself never sees the sphere's true position, radius, or color.
IMG = 16
SPHERE_CENTER = np.array([0.0, 0.0, 0.0])
SPHERE_RADIUS = 1.0
SPHERE_COLOR = np.array([1.0, 0.3, 0.2])
def look_at(cam_pos, target=np.array([0, 0, 0]), up=np.array([0, 1, 0])):
z = (cam_pos - target); z = z / np.linalg.norm(z)
x = np.cross(up, z); x = x / np.linalg.norm(x)
y = np.cross(z, x)
return np.stack([x, y, z], axis=1)
def get_rays(cam_pos, R, img_size=IMG, fov=60.0):
f = img_size / (2 * np.tan(np.radians(fov) / 2))
ys, xs = np.meshgrid(np.arange(img_size), np.arange(img_size), indexing='ij')
dirs_cam = np.stack([(xs - img_size/2 + 0.5) / f, -(ys - img_size/2 + 0.5) / f,
-np.ones_like(xs, dtype=np.float64)], axis=-1)
dirs_world = dirs_cam @ R.T
dirs_world = dirs_world / np.linalg.norm(dirs_world, axis=-1, keepdims=True)
origins = np.broadcast_to(cam_pos, dirs_world.shape)
return origins.astype(np.float32), dirs_world.astype(np.float32)
def analytic_render(cam_pos):
R = look_at(cam_pos)
origins, dirs = get_rays(cam_pos, R)
oc = origins - SPHERE_CENTER
a = np.sum(dirs * dirs, axis=-1)
b = 2 * np.sum(oc * dirs, axis=-1)
c = np.sum(oc * oc, axis=-1) - SPHERE_RADIUS ** 2
disc = b ** 2 - 4 * a * c
hit = disc > 0
t = np.where(hit, (-b - np.sqrt(np.clip(disc, 0, None))) / (2 * a), np.inf)
hit = hit & (t > 0)
img = np.zeros((IMG, IMG, 3), dtype=np.float32)
hit_pos = origins + np.where(hit, t, 0)[..., None] * dirs
normal = (hit_pos - SPHERE_CENTER) / SPHERE_RADIUS
light_dir = np.array([1.0, 1.0, 1.0]); light_dir = light_dir / np.linalg.norm(light_dir)
shade = np.clip(np.sum(normal * light_dir, axis=-1), 0.2, 1.0)
img[hit] = (SPHERE_COLOR[None, :] * shade[..., None])[hit]
return img, origins, dirs, hit
n_train_views = 8
angles = np.linspace(0, 2 * np.pi, n_train_views, endpoint=False)
cam_positions = [np.array([3 * np.cos(a), 1.0, 3 * np.sin(a)]) for a in angles]
train_images, train_origins, train_dirs = [], [], []
for cp in cam_positions:
img, origins, dirs, hit = analytic_render(cp)
train_images.append(img); train_origins.append(origins); train_dirs.append(dirs)
train_images = torch.tensor(np.array(train_images))
train_origins = torch.tensor(np.array(train_origins))
train_dirs = torch.tensor(np.array(train_dirs))
# a held-out test view, exactly BETWEEN two training camera positions
test_angle = angles[0] + (angles[1] - angles[0]) / 2
test_cam = np.array([3 * np.cos(test_angle), 1.0, 3 * np.sin(test_angle)])
test_img, test_origins, test_dirs, test_hit = analytic_render(test_cam)
test_img_t, test_origins_t, test_dirs_t = torch.tensor(test_img), torch.tensor(test_origins), torch.tensor(test_dirs)
fig, axes = plt.subplots(1, 5, figsize=(11, 2.5))
for ax, im in zip(axes[:4], train_images[:4]):
ax.imshow(im.numpy()); ax.axis('off')
axes[4].imshow(test_img); axes[4].axis('off'); axes[4].set_title('held-out\n(never trained on)', fontsize=8)
axes[0].set_title('training views', fontsize=8, loc='left')
plt.show()
The scene as a network: coordinates in, color and density out¶
A NeRF is a small MLP: feed it a 3D point, it returns a color and a density (how opaque the scene is at that point — near 0 in empty space, large inside solid material). Raw (x, y, z) coordinates are first expanded through a sinusoidal positional encoding at several frequencies (exactly Lesson 45's positional encoding, applied to 3D position instead of sequence position) — an MLP fed raw low-dimensional coordinates struggles to represent sharp, high-frequency detail, and this encoding fixes that.
class TinyNeRF(nn.Module):
def __init__(self, hidden=64, n_freqs=6):
super().__init__()
self.n_freqs = n_freqs
in_dim = 3 * (2 * n_freqs + 1)
self.net = nn.Sequential(
nn.Linear(in_dim, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU(),
nn.Linear(hidden, 4), # rgb (3) + density (1)
)
# bootstrap: bias the initial density prediction upward so early gradients
# actually reach the color head, instead of vanishing behind near-zero density
with torch.no_grad():
self.net[-1].bias[3] = 1.0
def encode(self, x):
out = [x]
for f in range(self.n_freqs):
out.append(torch.sin(2 ** f * np.pi * x))
out.append(torch.cos(2 ** f * np.pi * x))
return torch.cat(out, dim=-1)
def forward(self, x):
out = self.net(self.encode(x))
rgb = torch.sigmoid(out[..., :3])
sigma = F.softplus(out[..., 3])
return rgb, sigma
Volumetric rendering: turning (color, density) samples into a pixel¶
For each pixel's ray, sample points along it, evaluate the network at each, and composite the colors weighted by how much of the ray's "light" survives to reach that point — the same alpha-blending idea as Lesson 2's cv2.addWeighted, but chained across many samples along a ray instead of two whole images: each point's density determines how much it blocks the points behind it. This entire computation is differentiable end to end, so gradients flow from a rendered pixel's error all the way back into the network's weights.
def render_rays(model, origins, dirs, n_samples=32, near=1.0, far=5.0):
t_vals = torch.linspace(near, far, n_samples)
pts = origins[..., None, :] + dirs[..., None, :] * t_vals[:, None] # (..., n_samples, 3)
rgb, sigma = model(pts)
delta = t_vals[1:] - t_vals[:-1]
delta = torch.cat([delta, torch.tensor([1e10])])
alpha = 1.0 - torch.exp(-sigma * delta)
trans = torch.cumprod(torch.cat([torch.ones_like(alpha[..., :1]), 1.0 - alpha + 1e-10], dim=-1), dim=-1)[..., :-1]
weights = alpha * trans # how much each sample actually contributes to the final pixel
return (weights[..., None] * rgb).sum(dim=-2)
torch.manual_seed(0)
model = TinyNeRF()
opt = torch.optim.Adam(model.parameters(), lr=0.005)
V = train_images.shape[0]
origins_flat = train_origins.reshape(V, -1, 3)
dirs_flat = train_dirs.reshape(V, -1, 3)
images_flat = train_images.reshape(V, -1, 3)
for epoch in range(800):
v = np.random.default_rng(epoch).integers(V)
idx = np.random.default_rng(epoch + 1000).permutation(origins_flat.shape[1]) # all 256 pixels, shuffled
o, d, target = origins_flat[v, idx], dirs_flat[v, idx], images_flat[v, idx]
pred = render_rays(model, o, d)
loss = F.mse_loss(pred, target)
opt.zero_grad()
loss.backward()
opt.step()
print(f'final training loss: {loss.item():.4f}')
Novel view synthesis: rendering a camera position never seen in training¶
The real test isn't how well the network reproduces the 8 training photos — it's whether it can render a new camera position exactly halfway between two training views, which it never received a single pixel of supervision for.
with torch.no_grad():
pred_test = render_rays(model, test_origins_t.reshape(-1, 3), test_dirs_t.reshape(-1, 3)).reshape(IMG, IMG, 3)
mse = F.mse_loss(pred_test, test_img_t).item()
psnr = -10 * np.log10(mse) if mse > 0 else float('inf')
mean_color = train_images.reshape(-1, 3).mean(dim=0)
baseline_mse = F.mse_loss(mean_color[None, None, :].expand(IMG, IMG, 3), test_img_t).item()
print(f'held-out test view MSE: {mse:.4f} (PSNR = {psnr:.1f} dB)')
print(f'baseline (mean training color everywhere) MSE: {baseline_mse:.4f}')
fig, axes = plt.subplots(1, 2, figsize=(6, 3))
axes[0].imshow(test_img); axes[0].set_title('ground truth\n(held-out view)', fontsize=9); axes[0].axis('off')
axes[1].imshow(pred_test.clamp(0, 1).numpy()); axes[1].set_title('NeRF rendering', fontsize=9); axes[1].axis('off')
plt.show()
The rendered novel view beats the trivial mean-color baseline by a clear margin, and visibly reconstructs the sphere's shading and silhouette from a camera angle it never trained on — the network has implicitly learned the sphere's 3D geometry (where it is, how big it is) purely from 2D photographic evidence and the geometric constraint that every ray's color must be explained consistently across all 8 views simultaneously.
3D Gaussian Splatting: the same problem, a different representation¶
3D Gaussian Splatting (Kerbl et al., 2023) solves the identical problem — dense, photorealistic novel-view synthesis from posed images — with an explicit scene representation instead of NeRF's implicit MLP: the scene is a large collection of 3D Gaussian "blobs," each with a position, size/orientation, color, and opacity, directly optimized (not queried through a network) to reproduce the training photos when splatted (projected and alpha-blended, the same compositing math this lesson just implemented) onto each camera view. The tradeoff is speed: rendering an MLP requires a full forward pass per sample point along every ray, while rendering a fixed set of Gaussians is a much cheaper rasterization operation — closer to classical graphics — which is why Gaussian Splatting can render in real time where NeRF historically couldn't. Both approaches optimize purely against 2D photometric loss, with no 3D ground truth ever in the loop; they differ in what gets optimized, not in how the training signal reaches it.
Exercise¶
- Reduce
n_train_viewsfrom 8 to 4. Does the held-out view's PSNR drop substantially — and does the rendered image show visible artifacts on the side of the sphere that's now farther from any training camera? - Increase
n_samplesinrender_raysfrom 32 to 8. Does rendering quality degrade, and can you see why too few samples along a ray would fail to capture where the sphere's surface actually is? - This lesson's scene is a single diffuse (Lambertian) sphere, so the network only ever needs to learn
position -> color, density, never(position, view direction) -> color. Real NeRF also conditions color on viewing direction, to capture view-dependent effects like specular highlights. Sketch how you'd changeTinyNeRF.forwardto take a view direction as a second input, and describe a material (e.g. a shiny sphere) where this would visibly matter and this lesson's diffuse sphere wouldn't.
Closing the course¶
This lesson closes Part 4, and with it the course: from raw pixels and convolution (Part 1) through classical 3D geometry (Part 2), learned convolutional features (Part 3), and attention and self-supervision at scale (Part 4), neural rendering is a fitting endpoint because it ties the whole arc together in one place — Part 2's projective geometry and image formation, Part 3's convolutional feature learning, and Part 4's habit of replacing a hand-designed algorithm with a differentiable one trained purely against pixels, all combined to reconstruct a 3D scene from nothing but 2D photographs and the constraint that they must be mutually consistent.