Lesson 42: Instance Segmentation¶
Semantic segmentation (Lesson 41) answers "which pixels are circle pixels?" It has no notion of how many circles there are — two touching circles are just one connected blob of "circle" pixels. Instance segmentation answers the harder question: which pixels belong to this specific object, as opposed to that other object of the same class. This lesson shows the failure concretely, then fixes it with an idea that traces straight back to Lesson 12's Hough transform: instead of only classifying pixels, have the network also predict, for every foreground pixel, a vote for where its object's center is — then cluster the votes to recover individual instances, the same accumulator-and-peak-finding pattern used for lines and circles, now finding object centers instead.
import numpy as np
import cv2
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
Overlapping instances¶
Two circles per scene, deliberately placed close enough to touch or overlap. Ground truth includes both a semantic mask (0=background, 1=circle) and an instance mask (0=background, 1=first circle, 2=second circle).
SIZE = 32
def make_scene(rng, size=SIZE, n_circles=2, r=5):
scene = np.zeros((size, size), dtype=np.float32)
sem_mask = np.zeros((size, size), dtype=np.int64)
inst_mask = np.zeros((size, size), dtype=np.int64)
centers = []
for k in range(n_circles):
if k == 0:
cx, cy = rng.integers(r + 8, size - r - 8), rng.integers(r + 8, size - r - 8)
else:
px, py = centers[-1]
angle = rng.uniform(0, 2 * np.pi)
dist = rng.uniform(6, 9) # overlapping (< 2r) but centers still separable
cx = int(np.clip(px + dist * np.cos(angle), r, size - r - 1))
cy = int(np.clip(py + dist * np.sin(angle), r, size - r - 1))
centers.append((cx, cy))
yy, xx = np.mgrid[0:size, 0:size]
m = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r ** 2
scene[m] = 1.0
sem_mask[m] = 1
inst_mask[m] = k + 1 # later circles paint over earlier ones at overlaps
scene = np.clip(scene + rng.normal(0, 0.05, scene.shape), 0, 1).astype(np.float32)
return scene, sem_mask, inst_mask, centers
rng = np.random.default_rng(17)
N = 300
scenes, sem_masks, inst_masks, all_centers = [], [], [], []
for _ in range(N):
s, sm, im, c = make_scene(rng)
scenes.append(s); sem_masks.append(sm); inst_masks.append(im); all_centers.append(c)
scenes = np.array(scenes, dtype=np.float32)
sem_masks = np.array(sem_masks, dtype=np.int64)
fig, axes = plt.subplots(2, 4, figsize=(9, 4.5))
for i in range(4):
axes[0, i].imshow(scenes[i], cmap='gray'); axes[0, i].axis('off')
axes[1, i].imshow(inst_masks[i], cmap='viridis'); axes[1, i].axis('off')
axes[0, 0].set_title('input', fontsize=9, loc='left')
axes[1, 0].set_title('true instance mask', fontsize=9, loc='left')
plt.show()
Why semantic segmentation isn't enough¶
Train Lesson 41's exact U-Net architecture on the semantic task (background vs. circle) only, then try to recover instances the "obvious" way: run connected-components on the predicted foreground mask.
# build offset targets: for every foreground pixel, the (dx, dy) to ITS instance's center
yy, xx = np.mgrid[0:SIZE, 0:SIZE]
offset_targets = np.zeros((N, 2, SIZE, SIZE), dtype=np.float32)
for i in range(N):
im = inst_masks[i]
for k, (cx, cy) in enumerate(all_centers[i]):
m = im == (k + 1)
offset_targets[i, 0][m] = (cx - xx[m]) / SIZE
offset_targets[i, 1][m] = (cy - yy[m]) / SIZE
split = int(0.85 * N)
Xtr, Str, Otr = scenes[:split], sem_masks[:split], offset_targets[:split]
Xte, Ste, Ote = scenes[split:], sem_masks[split:], offset_targets[split:]
inst_te, centers_te = inst_masks[split:], all_centers[split:]
class InstanceNet(nn.Module):
"""Lesson 41's U-Net, with a second head: per-pixel offset-to-center regression"""
def __init__(self):
super().__init__()
self.enc1 = nn.Sequential(nn.Conv2d(1, 16, 3, padding=1), nn.ReLU())
self.enc2 = nn.Sequential(nn.Conv2d(16, 32, 3, padding=1), nn.ReLU())
self.pool = nn.MaxPool2d(2)
self.up = nn.Upsample(scale_factor=2, mode='nearest')
self.dec1 = nn.Sequential(nn.Conv2d(32 + 16, 16, 3, padding=1), nn.ReLU())
self.sem_head = nn.Conv2d(16, 2, 1) # background vs. circle
self.offset_head = nn.Conv2d(16, 2, 1) # (dx, dy) to this pixel's instance center
def forward(self, x):
f1 = self.enc1(x)
f2 = self.enc2(self.pool(f1))
d1 = self.dec1(torch.cat([self.up(f2), f1], dim=1))
return self.sem_head(d1), self.offset_head(d1)
torch.manual_seed(0)
model = InstanceNet()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
Xt = torch.tensor(Xtr).unsqueeze(1); St = torch.tensor(Str); Ot = torch.tensor(Otr)
fg_mask_t = (St == 1).unsqueeze(1).float()
for _ in range(300):
opt.zero_grad()
sem_logits, offset_pred = model(Xt)
sem_loss = F.cross_entropy(sem_logits, St)
offset_loss = (F.mse_loss(offset_pred, Ot, reduction='none') * fg_mask_t).sum() / fg_mask_t.sum().clamp(min=1)
(sem_loss + 2.0 * offset_loss).backward()
opt.step()
with torch.no_grad():
sem_logits_te, offset_pred_te = model(torch.tensor(Xte).unsqueeze(1))
sem_preds = sem_logits_te.argmax(1).numpy()
offset_preds = offset_pred_te.numpy()
print(f'semantic pixel accuracy: {(sem_preds == Ste).mean():.1%}')
n_cc_correct = 0
for i in range(len(Xte)):
binary_mask = (sem_preds[i] == 1).astype(np.uint8)
n_components, _ = cv2.connectedComponents(binary_mask)
n_components -= 1 # subtract the background label
true_n = len(set(inst_te[i].ravel()) - {0})
if n_components == true_n:
n_cc_correct += 1
print(f'connected-components recovers the correct instance count: {n_cc_correct}/{len(Xte)} scenes '
f'({n_cc_correct/len(Xte):.1%})')
Fix: vote for the center, then cluster (Hough, again)¶
The second head above was already trained to predict, for every foreground pixel, an offset pointing toward its own instance's center — supervised directly from the ground-truth instance masks. Each foreground pixel casts one vote (its predicted center location) into an accumulator, exactly like Lesson 12's Hough line/circle voting. Because every pixel belonging to the same circle votes for approximately the same point, the accumulator forms one tight cluster of votes per instance, however tangled the pixels themselves are. Finding instances becomes: find the vote clusters, then assign each pixel to its nearest cluster.
def cluster_instances(binary_mask, offset_pred, size=SIZE, peak_dist=4, vote_thresh=3):
ys, xs = np.where(binary_mask)
if len(xs) == 0:
return np.zeros_like(binary_mask, dtype=np.int64), []
pred_cx = xs + offset_pred[0][ys, xs] * size
pred_cy = ys + offset_pred[1][ys, xs] * size
votes = np.zeros((size, size), dtype=np.float32)
for cx, cy in zip(pred_cx, pred_cy):
cxi = int(np.clip(round(cx), 0, size - 1)); cyi = int(np.clip(round(cy), 0, size - 1))
votes[cyi, cxi] += 1
# greedily take the highest-voted peak, suppress its neighborhood, repeat
# (the same non-max-suppression idea as Lesson 39's detection boxes)
peaks = []
votes_work = votes.copy()
for _ in range(6):
idx = np.unravel_index(votes_work.argmax(), votes_work.shape)
if votes_work[idx] < vote_thresh:
break
peaks.append((idx[1], idx[0]))
y0, y1 = max(0, idx[0] - peak_dist), min(size, idx[0] + peak_dist + 1)
x0, x1 = max(0, idx[1] - peak_dist), min(size, idx[1] + peak_dist + 1)
votes_work[y0:y1, x0:x1] = 0
if not peaks:
return np.zeros_like(binary_mask, dtype=np.int64), []
inst_pred = np.zeros_like(binary_mask, dtype=np.int64)
peaks_arr = np.array(peaks)
for y, x in zip(ys, xs):
dists = (peaks_arr[:, 0] - x) ** 2 + (peaks_arr[:, 1] - y) ** 2
inst_pred[y, x] = np.argmin(dists) + 1
return inst_pred, peaks
n_correct = 0
inst_preds, all_peaks = [], []
for i in range(len(Xte)):
binary_mask = sem_preds[i] == 1
inst_pred, peaks = cluster_instances(binary_mask, offset_preds[i])
inst_preds.append(inst_pred); all_peaks.append(peaks)
true_n = len(set(inst_te[i].ravel()) - {0})
if len(peaks) == true_n:
n_correct += 1
print(f'offset-voting recovers the correct instance count: {n_correct}/{len(Xte)} scenes '
f'({n_correct/len(Xte):.1%})')
print(f'(vs. {n_cc_correct}/{len(Xte)} for connected components: {n_cc_correct/len(Xte):.1%})')
fig, axes = plt.subplots(3, 4, figsize=(9, 6.5))
for i in range(4):
axes[0, i].imshow(scenes[split + i], cmap='gray'); axes[0, i].axis('off')
axes[1, i].imshow(inst_te[i], cmap='viridis'); axes[1, i].axis('off')
axes[2, i].imshow(inst_preds[i], cmap='viridis'); axes[2, i].axis('off')
for r, name in enumerate(['input', 'true instances', 'recovered via offset voting']):
axes[r, 0].set_title(name, fontsize=9, loc='left')
plt.tight_layout()
plt.show()
Mask R-CNN, and the three flavors of "segmentation"¶
The offset-voting approach in this lesson is one real family of instance segmentation methods (related to techniques sometimes called "instance embedding" or center-voting). The other dominant approach, Mask R-CNN (He et al., 2017★), takes a more direct route: extend a two-stage detector (Lesson 40's R-CNN family) with a third output per detected box — alongside the existing class label and refined box coordinates, predict a small binary mask within that box. Detection already solves the "how many objects, and roughly where" problem via region proposals and NMS; Mask R-CNN just adds "and here's this one's exact silhouette."
Putting Lessons 37, 40, 41, and this lesson together, there are three distinct tasks that are easy to conflate:
- Classification (Lesson 37): one label for the whole image.
- Semantic segmentation (Lesson 41): one label per pixel, with no notion of separate objects of the same class.
- Instance segmentation (this lesson): one label per pixel, and a distinct identity for every individual object — "background," "circle #1," "circle #2," not just "background," "circle."
(A fourth term, panoptic segmentation, unifies the last two: every pixel gets both a semantic class and, for countable "thing" classes like circles or cars, an instance ID — while uncountable "stuff" classes like sky or road are labeled semantically only, since instance identity doesn't make sense for them.)
Exercise¶
- Reduce
distinmake_scenefrom the range(6, 9)to(2, 5), making the circles overlap much more heavily (centers closer together). Does the offset-voting method's instance-count accuracy hold up, or does it start failing too — and does the failure mode look different from connected-components' failure? - The clustering here uses
peak_dist=4for non-max suppression on the vote accumulator. Trypeak_dist=8. Does accuracy improve, get worse, or become sensitive to exactly how close two true circle centers happen to be in a given scene? - This lesson only ever has 2 circles per scene. Extend
make_sceneto place a random number of circles (1 to 4) and adjustcluster_instances's loop bound accordingly. Does the offset-voting approach's instance count accuracy hold up as the true number of instances grows?