Lesson 40: Object Detection II — From Classification to Regression¶

Lesson 39's sliding-window detector classified thousands of fixed-size windows and merged the survivors with NMS. That works, but it is fundamentally a classification approach bolted onto a search — the network never predicts a box directly, only "face or not, at this exact window." Modern detectors instead treat localization as a regression problem: given an image, directly predict box coordinates. This lesson builds the simplest possible version of that idea, then surveys how real detectors (R-CNN, YOLO, SSD) scale it up.

In [1]:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import matplotlib.patches as patches

One object per image: predict a box directly¶

The simplest possible regression detector: one circular blob per image, at an unknown location and size. Instead of a class label, the network's target is now four numbers — (x0, y0, width, height) of the bounding box, normalized to [0, 1] by image size.

In [2]:

SIZE = 32

def make_scene(rng, size=SIZE, obj_size=8):
    scene = np.zeros((size, size), dtype=np.float32)
    cx = rng.integers(obj_size, size - obj_size)
    cy = rng.integers(obj_size, size - obj_size)
    yy, xx = np.mgrid[0:size, 0:size]
    scene[((xx - cx) ** 2 + (yy - cy) ** 2) <= (obj_size * 0.5) ** 2] = 1.0
    scene = np.clip(scene + rng.normal(0, 0.05, scene.shape), 0, 1).astype(np.float32)
    box = (cx - obj_size // 2, cy - obj_size // 2, obj_size, obj_size)  # x0, y0, w, h
    return scene, box

rng = np.random.default_rng(9)
N = 400
scenes, boxes = [], []
for _ in range(N):
    s, b = make_scene(rng)
    scenes.append(s); boxes.append(b)
scenes = np.array(scenes, dtype=np.float32)
boxes = np.array(boxes, dtype=np.float32)

split = int(0.85 * N)
Xtr, Btr = scenes[:split], boxes[:split] / SIZE
Xte, Bte = scenes[split:], boxes[split:] / SIZE

fig, axes = plt.subplots(1, 4, figsize=(9, 2.5))
for ax, im, b in zip(axes, Xtr[:4], boxes[:4]):
    ax.imshow(im, cmap='gray')
    ax.add_patch(patches.Rectangle((b[0], b[1]), b[2], b[3], edgecolor='lime', facecolor='none', linewidth=2))
    ax.axis('off')
plt.show()

No description has been provided for this image

The model and the IoU metric¶

The network is a CNN backbone (Lesson 33's pattern) followed by a 4-output regression head with a sigmoid, so every prediction lands in [0, 1] — a valid normalized box coordinate. It's trained with plain MSE loss against the true box, and evaluated with IoU (Lesson 39's intersection-over-union), the metric that actually matters for detection: how much the predicted and true boxes overlap, not how close the four numbers are in isolation.

In [3]:

class Detector(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 16, 5, padding=2), nn.ReLU(), nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 5, padding=2), nn.ReLU(), nn.AdaptiveMaxPool2d(1),
        )
        self.fc = nn.Linear(32, 4)  # x0, y0, w, h, normalized

    def forward(self, x):
        return torch.sigmoid(self.fc(self.conv(x).flatten(1)))

def iou_batch(pred, target):
    px0, py0, pw, ph = pred[:, 0], pred[:, 1], pred[:, 2], pred[:, 3]
    tx0, ty0, tw, th = target[:, 0], target[:, 1], target[:, 2], target[:, 3]
    px1, py1, tx1, ty1 = px0 + pw, py0 + ph, tx0 + tw, ty0 + th
    ix0, iy0 = torch.maximum(px0, tx0), torch.maximum(py0, ty0)
    ix1, iy1 = torch.minimum(px1, tx1), torch.minimum(py1, ty1)
    inter = (ix1 - ix0).clamp(min=0) * (iy1 - iy0).clamp(min=0)
    union = pw * ph + tw * th - inter
    return inter / union.clamp(min=1e-8)

torch.manual_seed(0)
model = Detector()
opt = torch.optim.Adam(model.parameters(), lr=0.005)
Xt = torch.tensor(Xtr).unsqueeze(1); Bt = torch.tensor(Btr)
for _ in range(400):
    opt.zero_grad()
    loss = F.mse_loss(model(Xt), Bt)
    loss.backward()
    opt.step()

with torch.no_grad():
    pred_te = model(torch.tensor(Xte).unsqueeze(1))
    ious = iou_batch(pred_te, torch.tensor(Bte))

print(f'mean IoU on test set: {ious.mean().item():.3f}')
print(f'fraction of test boxes with IoU > 0.5: {(ious > 0.5).float().mean().item():.1%}')

mean IoU on test set: 0.722
fraction of test boxes with IoU > 0.5: 90.0%

In [4]:

fig, axes = plt.subplots(1, 4, figsize=(9, 2.5))
for i, ax in enumerate(axes):
    ax.imshow(Xte[i], cmap='gray')
    tb = Bte[i] * SIZE
    pb = pred_te[i].numpy() * SIZE
    ax.add_patch(patches.Rectangle((tb[0], tb[1]), tb[2], tb[3], edgecolor='lime', facecolor='none', linewidth=2, label='true'))
    ax.add_patch(patches.Rectangle((pb[0], pb[1]), pb[2], pb[3], edgecolor='red', facecolor='none', linewidth=1.5, linestyle='--', label='pred'))
    ax.set_title(f'IoU={ious[i]:.2f}', fontsize=9)
    ax.axis('off')
axes[0].legend(fontsize=6, loc='upper left')
plt.show()

Scaling this up: two families of real detectors¶

This lesson's detector only handles exactly one object per image, because a fixed-size output vector (4 numbers) can only ever describe one box. Real scenes have a variable, unknown number of objects. Two different fixes became the two dominant families of detector:

Two-stage (R-CNN family): first generate a modest number of region proposals — candidate boxes likely to contain something, via a cheap, class-agnostic method (the original R-CNN used classical segmentation; Faster R-CNN (Ren et al., 2015★) learns a small "region proposal network" instead) — then run a classifier-plus-box-regressor (this lesson's whole architecture) on each proposal independently, exactly like running the sliding-window classifier from Lesson 39 but only at a handful of promising locations instead of every window. Accurate, but only as fast as (proposals) x (one forward pass) allows.

Single-stage (YOLO, SSD): skip proposals entirely. YOLO (You Only Look Once, Redmon et al., 2016★) divides the image into a coarse grid of cells, and has each grid cell directly predict (as this lesson's network does) a fixed number of boxes plus a class label plus a confidence score, all in one forward pass. To let a single cell describe objects of different aspect ratios, single-stage detectors use anchor boxes: several predefined box shapes (tall, wide, square) per cell, with the network predicting an offset from each anchor rather than a box from scratch. Faster to run, historically somewhat less accurate than two-stage methods, though the gap has narrowed considerably.

Both families end with the same postprocessing step this lesson skipped by only ever predicting one box: NMS (Lesson 39) to merge the overlapping candidate boxes any real multi-object scene produces.

Exercise¶

Change obj_size in make_scene from a fixed 8 to a random value (e.g. rng.integers(4, 12)) so objects vary in size, and retrain. Does mean IoU hold up, get worse, or barely change — and why would variable object scale be harder for a single fixed-size regression head than variable position?
The loss function here is plain MSE on (x0, y0, w, h), but the metric that matters is IoU. Replace the loss with 1 - iou_batch(pred, target).mean() (directly optimizing IoU) and compare final mean test IoU to the MSE-trained version. Real detectors (e.g. Faster R-CNN, YOLO variants) do exactly this with generalized IoU losses — can you see why MSE loss and IoU metric might disagree on which of two similar predictions is "better"?
This lesson's detector has no anchor boxes and only ever handles one object. Sketch (in words) how you would modify the architecture's output to handle up to 3 objects per image using a 3x3 grid where each grid cell predicts one box and a "there's an object here" confidence — the core idea behind YOLO's grid.

← Lesson 39: Object Detection I — Sliding Windows, from Rowley-Baluja-Kanade to Viola-Jones

Lesson 41: Semantic Segmentation →