Lesson 39: Visualizing and Understanding CNNs¶
A trained CNN is a black box because its millions of weights don't have obvious individual meanings. But where in the input image a prediction comes from is answerable, and answering it is often what separates "the model got the right answer" from "the model got the right answer for the right reason." This lesson builds two visualization tools from scratch — saliency maps (Simonyan et al., 2013) and Grad-CAM (Selvaraju et al., 2017★) — on real photos from CIFAR-10 (Krizhevsky, 2009), then uses them to catch a model that's cheating. Finally, t-SNE (Van der Maaten & Hinton, 2008★) is used to visualize the learned feature vectors.
import pickle
import tarfile
import urllib.request
from pathlib import Path
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
Setup: a cat-vs-automobile CNN, with feature maps exposed¶
A CIFAR-10 binary task: cat vs. automobile, 300 training images per class. The architecture is Lesson 34's CNN pattern, except forward now also returns the last convolutional layer's feature map (before global pooling), so both visualization methods have access to it.
CIFAR_URL = 'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'
CACHE_ROOT = Path.home() / '.cache' / 'cvintro'
CACHE_DIR = CACHE_ROOT / 'cifar-10-batches-py'
def ensure_cifar10():
if CACHE_DIR.exists():
return
CACHE_ROOT.mkdir(parents=True, exist_ok=True)
archive_path = CACHE_ROOT / 'cifar-10-python.tar.gz'
if not archive_path.exists():
print('Downloading CIFAR-10 (~163 MB, one-time, cached under ~/.cache/cvintro)...')
urllib.request.urlretrieve(CIFAR_URL, archive_path)
print('Extracting...')
with tarfile.open(archive_path) as tar:
tar.extractall(CACHE_ROOT)
def load_cifar_batch(path):
with open(path, 'rb') as f:
d = pickle.load(f, encoding='bytes')
imgs = d[b'data'].reshape(-1, 3, 32, 32).transpose(0, 2, 3, 1).astype(np.float32) / 255.0
labels = np.array(d[b'labels'], dtype=np.int64)
return imgs, labels
ensure_cifar10()
CIFAR_LABELS = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
CID = {name: CIFAR_LABELS.index(name) for name in CIFAR_LABELS}
train_imgs, train_labels = [], []
for i in range(1, 6):
imgs, labels = load_cifar_batch(CACHE_DIR / f'data_batch_{i}')
train_imgs.append(imgs); train_labels.append(labels)
train_imgs, train_labels = np.concatenate(train_imgs), np.concatenate(train_labels)
test_imgs, test_labels = load_cifar_batch(CACHE_DIR / 'test_batch')
def take(imgs, labels, name, n, rng_local):
idx = np.where(labels == CID[name])[0]
idx = rng_local.permutation(idx)[:n]
return imgs[idx].copy()
rng = np.random.default_rng(4)
X_cat_train = take(train_imgs, train_labels, 'cat', 300, rng)
X_auto_train = take(train_imgs, train_labels, 'automobile', 300, rng)
X_cat_test = take(test_imgs, test_labels, 'cat', 100, rng)
X_auto_test = take(test_imgs, test_labels, 'automobile', 100, rng)
X_train = np.concatenate([X_cat_train, X_auto_train])
y_train = np.array([0.0] * 300 + [1.0] * 300, dtype=np.float32)
X_test = np.concatenate([X_cat_test, X_auto_test])
y_test = np.array([0.0] * 100 + [1.0] * 100, dtype=np.float32)
class CNN(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(3, 8, 5, padding=2)
self.pool = nn.MaxPool2d(2)
self.conv2 = nn.Conv2d(8, 16, 5, padding=2)
self.gpool = nn.AdaptiveMaxPool2d(1)
self.fc = nn.Linear(16, 1)
def forward(self, x):
f1 = F.relu(self.conv1(x)) # 32x32, full input resolution
f2 = F.relu(self.conv2(self.pool(f1))) # last conv feature map, downsampled to 16x16
feat = self.gpool(f2).flatten(1)
return self.fc(feat).squeeze(-1), f2
torch.manual_seed(0)
model_original = CNN()
opt = torch.optim.Adam(model_original.parameters(), lr=0.001)
Xt = torch.tensor(X_train).permute(0, 3, 1, 2); yt = torch.tensor(y_train)
for _ in range(300):
opt.zero_grad()
out, _ = model_original(Xt)
loss = F.binary_cross_entropy_with_logits(out, yt)
loss.backward()
opt.step()
with torch.no_grad():
out, _ = model_original(torch.tensor(X_test).permute(0, 3, 1, 2))
acc = ((out > 0).float() == torch.tensor(y_test)).float().mean().item()
print(f'test accuracy: {acc:.1%}')
Saliency maps¶
The idea of saliency maps (Simonyan et al., 2013): take the gradient of the predicted class score with respect to every input pixel. A pixel with a large-magnitude gradient is one where a small change would most change the prediction — i.e., a pixel the network is "looking at."
def saliency_map(model, img_hw3):
x = torch.tensor(img_hw3).permute(2, 0, 1).unsqueeze(0)
x.requires_grad_(True)
score, feat = model(x)
score.backward()
return x.grad[0].abs().amax(dim=0).numpy(), feat # max abs gradient across the 3 color channels
idx = 3
saliency, _ = saliency_map(model_original, X_test[idx])
fig, axes = plt.subplots(1, 2, figsize=(7, 3.2))
axes[0].imshow(X_test[idx])
axes[0].set_title('input image'); axes[0].axis('off')
axes[1].imshow(saliency, cmap='hot')
axes[1].set_title('saliency map')
axes[1].axis('off')
plt.show()
Grad-CAM¶
Raw saliency maps are pixel-level and tend to be noisy. Grad-CAM (Selvaraju et al., 2017★) instead operates on the feature maps of a chosen convolutional layer, typically the last convolutional layer. Although these feature maps are lower-resolution than a pixel-level saliency map (downsampled by the pooling layers before them), they tend to carry more class-discriminative semantic information than individual raw-pixel gradients do. The method:
- Compute the gradient of the target class score with respect to each feature-map channel.
- Globally average each channel's gradients over its spatial dimensions to obtain one importance weight per channel.
- Take the weighted sum of the feature-map channels and apply ReLU, retaining the positive contributions to the target class.
The resulting coarse heatmap has the spatial resolution of the selected convolutional layer; it can be upsampled to the input resolution for visualization.
def grad_cam(model, img_hw3, out_size=32):
x = torch.tensor(img_hw3).permute(2, 0, 1).unsqueeze(0)
x.requires_grad_(True)
score, feat = model(x)
feat.retain_grad()
score.backward()
weights = feat.grad[0].mean(dim=(1, 2)) # (channels,) importance per channel
cam = F.relu((weights[:, None, None] * feat[0]).sum(dim=0))
cam_up = F.interpolate(cam[None, None], size=(out_size, out_size), mode='bilinear', align_corners=False)
return cam_up[0, 0].detach().numpy()
cam = grad_cam(model_original, X_test[idx])
fig, axes = plt.subplots(1, 3, figsize=(10, 3.2))
axes[0].imshow(X_test[idx])
axes[0].set_title('input image'); axes[0].axis('off')
axes[1].imshow(saliency, cmap='hot')
axes[1].set_title('saliency map'); axes[1].axis('off')
axes[2].imshow(cam, cmap='hot')
axes[2].set_title('Grad-CAM'); axes[2].axis('off')
plt.show()
Do these maps actually track what the model uses?¶
To test whether these tools are useful, deliberately give the model a shortcut, and check whether they correctly catch it. Add a small, unmistakable 5x5 white marker to the top-left corner of every automobile training image only (never on cat images) — a stand-in for a real-world confound, like a watermark, a lab-specific artifact, or a capture-device quirk that happens to correlate with one class. Train a second model, model_shortcut, on this corrupted dataset, and compare it against the original model_original above (never exposed to any marker).
def add_marker(imgs):
out = imgs.copy()
out[:, 0:5, 0:5, :] = 1.0 # a stark 5x5 white square, top-left corner
return out
X_train_shortcut = np.concatenate([X_cat_train, add_marker(X_auto_train)])
X_auto_test_marked = add_marker(X_auto_test)
X_test_marked = np.concatenate([X_cat_test, X_auto_test_marked]) # marker present at test time too
torch.manual_seed(0)
model_shortcut = CNN()
opt2 = torch.optim.Adam(model_shortcut.parameters(), lr=0.001)
Xt_shortcut = torch.tensor(X_train_shortcut).permute(0, 3, 1, 2)
for _ in range(300):
opt2.zero_grad()
out, _ = model_shortcut(Xt_shortcut)
loss = F.binary_cross_entropy_with_logits(out, torch.tensor(y_train))
loss.backward()
opt2.step()
def acc_of(m, X, y):
with torch.no_grad():
out, _ = m(torch.tensor(X).permute(0, 3, 1, 2))
return ((out > 0).float() == torch.tensor(y)).float().mean().item()
print(f'{"":>16} {"clean test":>12} {"marked test":>13}')
print(f'{"model_original":>16} {acc_of(model_original, X_test, y_test):>11.1%} {acc_of(model_original, X_test_marked, y_test):>13.1%}')
print(f'{"model_shortcut":>16} {acc_of(model_shortcut, X_test, y_test):>11.1%} {acc_of(model_shortcut, X_test_marked, y_test):>13.1%}')
model_original's accuracy barely moves whether the marker is present or not — since it never learned to use it, it has nothing to lose when it's absent.
model_shortcut looks better than model_original when the marker is present, but it collapses when the marker is removed. It learned the shortcut "white corner square = automobile".
Now use Grad-CAM to check whether the visualization tool actually catches this. Since the marker's location is known exactly (rows 0-4, columns 0-4), the "distance to true center" check from a controlled synthetic dataset becomes: what fraction of test images does each model's Grad-CAM peak land inside that exact 5x5 region?
def peak_in_marker_frac(method, m, imgs):
count = 0
for img in imgs:
heatmap = saliency_map(m, img)[0] if method == 'saliency' else grad_cam(m, img)
peak = np.unravel_index(heatmap.argmax(), heatmap.shape) # (row, col)
if peak[0] < 5 and peak[1] < 5:
count += 1
return count / len(imgs)
rows = [f'{"model":>16} {"test images":>20} {"saliency":>10} {"Grad-CAM":>10}']
for name, m, imgs in [('model_original', model_original, X_auto_test_marked),
('model_shortcut', model_shortcut, X_auto_test_marked),
('model_shortcut', model_shortcut, X_auto_test)]:
label = 'marked' if imgs is not X_auto_test else 'clean'
sal_frac = peak_in_marker_frac('saliency', m, imgs)
cam_frac = peak_in_marker_frac('grad_cam', m, imgs)
rows.append(f'{name:>16} {label:>20} {sal_frac:>9.1%} {cam_frac:>10.1%}')
print('\n'.join(rows))
model_original (never trained on the marker) rarely lands either method's peak in that exact corner. model_shortcut is a different story: on marked images both tools are nearly perfect (saliency 96%, Grad-CAM 100%), but on clean images — where the marker was never added, yet model_shortcut still relies on it internally — Grad-CAM's peak still lands there 92% of the time versus only 56% for saliency. Grad-CAM pools gradients over an entire feature-map channel before localizing, which smooths out pixel-level noise and makes it the more trustworthy detector when the shortcut's literal trigger isn't visible in a given image; model_shortcut's internal machinery is anchored to that spot regardless of what's actually there, exactly the failure mode described at the top of this lesson.
The practical lesson: accuracy alone (Lesson 36) can't distinguish whether the model has learned the real signal or a shortcut correlated with it in the training data. Visualization tools such as saliency and Grad-CAM can — but only if you know to check, if you pick a tool sturdy enough to trust, and if you interpret the results with some skepticism about what else in the image might be drawing gradient attention.
Feature-space visualization: t-SNE¶
Saliency maps and Grad-CAM both answer a spatial question about one image at a time: where is the network looking? t-SNE (van der Maaten & Hinton, 2008★) asks a broader question across many images: does the network’s internal representation separate the classes it was trained to recognize? It takes each image’s 16-d feature vector—the representation fed directly to the classifier—and projects these vectors into 2D while preserving local neighborhoods. Unlike PCA (Lesson 6), which uses a linear projection, t-SNE emphasizes local structure, making it useful for checking whether images from the same class cluster together in feature space without using their labels during the projection.
def tsne(X, n_iter=500, perplexity=15.0, lr=100.0, seed=0):
rng = np.random.default_rng(seed)
n = X.shape[0]
sq_dists = ((X[:, None, :] - X[None, :, :]) ** 2).sum(-1)
# high-dimensional affinities P: for each point, binary-search a Gaussian bandwidth so its
# neighbor distribution has the target perplexity (an implicit "how many neighbors matter" knob)
target_entropy = np.log(perplexity)
P = np.zeros((n, n))
for i in range(n):
others = np.arange(n) != i
d_i = sq_dists[i, others]
lo, hi = 1e-4, 1e4
for _ in range(50):
beta = (lo + hi) / 2
p = np.exp(-d_i * beta)
p /= p.sum() + 1e-12
entropy = -np.sum(p * np.log(p + 1e-12))
if entropy > target_entropy:
lo = beta # entropy too high (too spread out) -> need a larger beta to sharpen it
else:
hi = beta
P[i, others] = p
P = (P + P.T) / (2 * n) # symmetrize into one joint distribution over pairs
P = np.maximum(P, 1e-12) * 4.0 # early exaggeration: temporarily inflate P so true neighbors clump together faster
# low-dimensional embedding Y, fit by gradient descent on KL(P || Q)
Y = rng.normal(0, 1e-2, (n, 2))
velocity = np.zeros_like(Y)
for it in range(n_iter):
if it == 100:
P /= 4.0 # turn off early exaggeration once clusters have separated
d = ((Y[:, None, :] - Y[None, :, :]) ** 2).sum(-1)
num = 1.0 / (1.0 + d) # Student-t kernel -- heavy tails let moderately-distant points repel more strongly than a Gaussian would, avoiding the "crowding problem"
np.fill_diagonal(num, 0)
Q = np.maximum(num / num.sum(), 1e-12)
coeff = (P - Q) * num
grad = 4 * (coeff[:, :, None] * (Y[:, None, :] - Y[None, :, :])).sum(1)
momentum = 0.5 if it < 100 else 0.8
velocity = momentum * velocity - lr * grad
Y = Y + velocity
return Y
def pooled_features(m, X):
with torch.no_grad():
_, f2 = m(torch.tensor(X).permute(0, 3, 1, 2))
return F.adaptive_max_pool2d(f2, 1).flatten(1).numpy() # same pooling model.gpool does internally
feat_trained = pooled_features(model_original, X_test)
torch.manual_seed(1) # a second network, same architecture, never trained -- the baseline
untrained_model = CNN()
feat_untrained = pooled_features(untrained_model, X_test)
Y_trained = tsne(feat_trained, seed=0)
Y_untrained = tsne(feat_untrained, seed=0)
fig, axes = plt.subplots(1, 2, figsize=(9, 4))
for ax, Y, title in zip(axes, [Y_untrained, Y_trained], ['untrained CNN features', 'trained CNN features']):
ax.scatter(*Y[:100].T, c='tab:blue', s=14, label='cat')
ax.scatter(*Y[100:].T, c='tab:orange', s=14, label='automobile')
ax.set_title(title, fontsize=9)
ax.legend(fontsize=7)
fig.suptitle("t-SNE of each test image's 16-d pooled feature vector, colored by true class", y=1.02)
plt.show()
The untrained network's features scatter the two classes together with no visible structure — unsurprising, since its convolutional weights are still random and have never seen a single labeled example. The trained network's features form two visibly separated clumps, cat and automobile, even though t-SNE itself was never told which point belonged to which class; it only saw the 16-d feature vectors and their pairwise distances. (Quantitatively, a simple 1-nearest-neighbor check in the 2D embedding — does each point's nearest neighbor share its true label? — reveals 78% for the trained features versus about 62% for the untrained ones.) This is evidence that training didn't just adjust the final linear layer — it reshaped the whole feature space so that cats and automobiles genuinely live in different neighborhoods of it.
Exercises¶
- Grad-CAM here uses the last conv layer. Modify
grad_camto instead use the intermediate feature map afterconv1(before the pool andconv2). Does the resulting heatmap get sharper (closer to pixel-perfect, like the saliency map) or coarser, and why would an earlier layer behave that way? - Shrink the marker from 5x5 to 2x2, or dim it from pure white (
1.0) to a faint gray (0.6). Doesmodel_shortcutstill learn to rely on it as strongly (check the clean-vs-marked accuracy gap), and does Grad-CAM still catch it as reliably? - The saliency-map gradient in this lesson is taken with respect to the raw logit (
score), not the sigmoid probability. Try computing it with respect totorch.sigmoid(score)instead — does the resulting map look meaningfully different, and can you explain why using the chain rule? - Color the trained-feature t-SNE plot by predicted label instead of true label (
(model_original(...) > 0)from earlier). Do the handful of points that land on the "wrong side" of the cluster boundary correspond to images the model actually misclassifies? Then repeat withmodel_shortcut's features onX_auto_test_marked— does removing the marker (X_auto_test, unmarked) collapse the clean class separation the way the earlier accuracy numbers predicted it would?