Lesson 37: Image Classification in Practice¶

Every classifier so far has been evaluated with a single number: overall accuracy. That number hides a lot. This lesson builds a 4-class classifier on a deliberately imbalanced dataset and shows why accuracy alone can be misleading, using the tools that reveal what's actually going wrong: the confusion matrix, and per-class precision and recall.

In [1]:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt

A 4-class, imbalanced dataset¶

Four noisy synthetic shapes (plus, circle, square, triangle). The test set has a roughly even mix of all four, but the training set is deliberately starved of triangles — a stand-in for the common real-world situation where some classes are just rarer to collect than others.

In [2]:

SHAPES = ['plus', 'circle', 'square', 'triangle']

def make_image(shape_type, cx, cy, size=16):
    img = np.zeros((size, size), dtype=np.float32)
    if shape_type == 'plus':
        img[cy-1:cy+2, cx-3:cx+4] = 1.0
        img[cy-3:cy+4, cx-1:cx+2] = 1.0
    elif shape_type == 'circle':
        yy, xx = np.mgrid[0:size, 0:size]
        img[((xx-cx)**2 + (yy-cy)**2) <= 9] = 1.0
    elif shape_type == 'square':
        img[cy-3:cy+4, cx-3:cx+4] = 1.0
    elif shape_type == 'triangle':
        yy, xx = np.mgrid[0:size, 0:size]
        h = 7
        mask = (yy >= cy - h//2) & (yy <= cy + h//2) & (np.abs(xx - cx) <= (yy - (cy - h//2)) * 0.8)
        img[mask] = 1.0
    return img

def make_dataset(rng_local, n, position_range=(4, 12), noise=0.5):
    imgs, labels = [], []
    for _ in range(n):
        shape_type = rng_local.choice(SHAPES)
        cx, cy = rng_local.integers(*position_range), rng_local.integers(*position_range)
        img = make_image(shape_type, cx, cy)
        img = np.clip(img + rng_local.normal(0, noise, img.shape), 0, 1).astype(np.float32)
        imgs.append(img)
        labels.append(SHAPES.index(shape_type))
    return np.array(imgs, dtype=np.float32), np.array(labels, dtype=np.int64)

rng = np.random.default_rng(7)
X_train, y_train = make_dataset(rng, 400)
X_test, y_test = make_dataset(rng, 200)

# downsample triangle in the training set only, to create class imbalance
mask = ~((y_train == 3) & (rng.random(len(y_train)) < 0.92))
X_train, y_train = X_train[mask], y_train[mask]

print('training class counts:', dict(zip(SHAPES, np.bincount(y_train))))
print('test class counts:    ', dict(zip(SHAPES, np.bincount(y_test))))

fig, axes = plt.subplots(1, 4, figsize=(8, 2.2))
for ax, name in zip(axes, SHAPES):
    ax.imshow(X_train[y_train == SHAPES.index(name)][0], cmap='gray')
    ax.set_title(name, fontsize=9)
    ax.axis('off')
plt.show()

training class counts: {'plus': np.int64(99), 'circle': np.int64(94), 'square': np.int64(104), 'triangle': np.int64(10)}
test class counts:     {'plus': np.int64(56), 'circle': np.int64(48), 'square': np.int64(45), 'triangle': np.int64(51)}

No description has been provided for this image

In [3]:

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 16, 5, padding=2), nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 5, padding=2), nn.ReLU(),
            nn.AdaptiveMaxPool2d(1),
        )
        self.fc = nn.Linear(32, 4)

    def forward(self, x):
        return self.fc(self.conv(x).flatten(1))

torch.manual_seed(0)
model = CNN()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
Xt = torch.tensor(X_train).unsqueeze(1); yt = torch.tensor(y_train)
for _ in range(300):
    opt.zero_grad()
    loss = F.cross_entropy(model(Xt), yt)
    loss.backward()
    opt.step()

Xte = torch.tensor(X_test).unsqueeze(1)
with torch.no_grad():
    preds = model(Xte).argmax(1).numpy()

acc = (preds == y_test).mean()
print(f'overall test accuracy: {acc:.1%}')

overall test accuracy: 82.0%

82% overall accuracy sounds fine on its own. It hides something important: the model is not equally good at all four classes.

Confusion matrix¶

Row i, column j counts test examples of true class i predicted as class j. A perfect classifier is diagonal; everything off the diagonal is a specific, nameable mistake.

In [4]:

cm = np.zeros((4, 4), dtype=int)
for t, p in zip(y_test, preds):
    cm[t, p] += 1

print('confusion matrix (rows=true, cols=predicted):')
print(f'{"":>10}' + ''.join(f'{s:>10}' for s in SHAPES))
for i, s in enumerate(SHAPES):
    print(f'{s:>10}' + ''.join(f'{cm[i, j]:>10}' for j in range(4)))

fig, ax = plt.subplots(figsize=(4.5, 4))
im = ax.imshow(cm, cmap='Blues')
ax.set_xticks(range(4)); ax.set_xticklabels(SHAPES, rotation=45)
ax.set_yticks(range(4)); ax.set_yticklabels(SHAPES)
ax.set_xlabel('predicted'); ax.set_ylabel('true')
for i in range(4):
    for j in range(4):
        ax.text(j, i, cm[i, j], ha='center', va='center',
                 color='white' if cm[i, j] > cm.max() / 2 else 'black')
plt.title('Confusion matrix')
plt.tight_layout()
plt.show()

confusion matrix (rows=true, cols=predicted):
                plus    circle    square  triangle
      plus        47         9         0         0
    circle         5        43         0         0
    square         0         0        45         0
  triangle        10         7         5        29

True positives, false positives, true negatives, false negatives¶

Every cell of the confusion matrix above is a specific kind of correctness or mistake, but the standard vocabulary for talking about them is binary: pick one class and ask only "is it this, or not?" Collapsing the 4-class matrix down to "triangle vs. everything else" gives exactly four outcomes:

True positive (TP): actually a triangle, predicted triangle. A hit.
False negative (FN): actually a triangle, predicted something else. A miss — the model let it slip past.
False positive (FP): actually not a triangle, predicted triangle anyway. A false alarm.
True negative (TN): actually not a triangle, correctly predicted not-triangle.

This is the same 2x2 table underlying every binary classifier's evaluation (a medical test's "positive/negative" result, a spam filter's "spam/not spam" decision) — a multi-class confusion matrix is just this table computed once per class, with everything off that class's row/column collapsed into "not this class."

In [5]:

cls = SHAPES.index('triangle')
tp = cm[cls, cls]
fn = cm[cls, :].sum() - tp          # true triangle, predicted something else
fp = cm[:, cls].sum() - tp          # predicted triangle, actually something else
tn = cm.sum() - tp - fn - fp        # everything else, correctly not called triangle

print(f'{"":>18}{"predicted triangle":>20}{"predicted NOT triangle":>24}')
print(f'{"actually triangle":>18}{tp:>20}{fn:>24}')
print(f'{"actually NOT triangle":>18}{fp:>20}{tn:>24}')
print()
print(f'TP={tp}, FP={fp}, FN={fn}, TN={tn}, total={tp+fp+fn+tn} (test set size={len(y_test)})')

                    predicted triangle  predicted NOT triangle
 actually triangle                  29                      22
actually NOT triangle                   0                     149

TP=29, FP=0, FN=22, TN=149, total=200 (test set size=200)

Precision and recall¶

Two numbers per class, computed directly from TP/FP/FN:

Recall = TP / (TP + FN) — of everything that really was class i, what fraction did the model catch? Low recall means the model misses that class often.
Precision = TP / (TP + FP) — of everything the model called class i, what fraction actually was? Low precision means the model cries wolf on that class often.

(A less commonly needed but related pair, built from the other two quadrants: specificity = TN / (TN + FP), how well the model avoids false alarms on the negative class, and its complement the false positive rate = FP / (FP + TN) = 1 − specificity.)

In [6]:

print(f'{"class":>10} {"precision":>10} {"recall":>8} {"support":>8}')
for i, name in enumerate(SHAPES):
    tp = cm[i, i]
    fn = cm[i, :].sum() - tp
    fp = cm[:, i].sum() - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else float('nan')
    recall = tp / (tp + fn) if (tp + fn) > 0 else float('nan')
    support = cm[i, :].sum()
    print(f'{name:>10} {precision:>10.2f} {recall:>8.2f} {support:>8}')

     class  precision   recall  support
      plus       0.76     0.84       56
    circle       0.73     0.90       48
    square       0.90     1.00       45
  triangle       1.00     0.57       51

Triangle — the class starved to just 10 training examples — has high precision but noticeably lower recall: when the model does say "triangle," it's usually right, but it fails to recognize a large fraction of the actual triangles, defaulting instead to whichever classes it saw plenty of during training. That is the standard signature of class imbalance, and it is completely invisible in the single overall-accuracy number from before. Plus and circle, by contrast, are both well-represented in training but get confused with each other — a different failure mode entirely, caused by genuine visual ambiguity between the two shapes under heavy noise rather than by a lack of data.

The practical lesson: always inspect the confusion matrix and per-class metrics before trusting a single accuracy figure, especially on any dataset where classes aren't naturally balanced.

Exercise¶

Reduce the noise level (noise=0.5 in make_dataset) to 0.2 and rerun. Does the plus/circle confusion mostly disappear? Does the triangle recall problem also improve, or does it persist — and why would data scarcity not be fixed by less noise?
Try weighting the loss by inverse class frequency (F.cross_entropy(logits, yt, weight=class_weights), where class_weights[i] = 1 / count(class i)) instead of downsampling triangles further. Does it recover triangle recall, and at what cost to the other classes' precision?
Compute the F1 score (the harmonic mean of precision and recall, 2 * p * r / (p + r)) for each class. Why might F1 be a better single number to track per-class than accuracy, when accuracy is only meaningful in aggregate?

← Lesson 36: Transfer Learning

Lesson 38: Visualizing and Understanding CNNs →