Open in Colab Download .ipynb

Lesson 31: Multi-Layer Perceptrons — Composing Projections

Lesson 30's single neuron, trained however carefully, could not beat chance on the ring-inside-a-disk dataset — a single linear projection just isn't expressive enough. This lesson adds one more layer: instead of one projection followed by a threshold, use two projections with a nonlinearity in between. That's it. That's the entire idea behind every deep network in this course — stack more of these.

In [1]:
import numpy as np
import torch
import matplotlib.pyplot as plt

The unsolvable problem, again

In [2]:
rng = np.random.default_rng(0)
theta_in = rng.uniform(0, 2 * np.pi, 60)
inner = np.stack([0.5 * np.cos(theta_in), 0.5 * np.sin(theta_in)], axis=1) + rng.normal(0, 0.1, (60, 2))
theta_out = rng.uniform(0, 2 * np.pi, 60)
outer = np.stack([2.0 * np.cos(theta_out), 2.0 * np.sin(theta_out)], axis=1) + rng.normal(0, 0.15, (60, 2))
X = np.vstack([inner, outer])
y = np.concatenate([np.zeros(60), np.ones(60)])

plt.scatter(*inner.T, s=15, label='inner (y=0)')
plt.scatter(*outer.T, s=15, label='outer (y=1)')
plt.legend(fontsize=8)
plt.gca().set_aspect('equal')
plt.title('No single linear projection separates these (Lessons 29-30)')
plt.show()
No description has been provided for this image

Two projections, one nonlinearity

A multi-layer perceptron (MLP) chains a hidden projection into a new space, a nonlinear activation applied elementwise, and then a final linear projection (exactly Lesson 30's neuron) on top of the transformed coordinates:

$$h = \text{ReLU}(W_1 x + b_1), \qquad z = w_2^\top h + b_2, \qquad p = \sigma(z)$$

$\text{ReLU}(u) = \max(0, u)$ is the simplest common activation: it's linear everywhere except a single kink at zero. That one kink, applied independently to every unit of $h$, is enough — the hidden layer can bend, fold, and stretch the input space so a linear boundary in the transformed space corresponds to a highly nonlinear boundary back in the original coordinates.

Backprop through two layers

The chain rule extends cleanly: propagate the error signal $\partial L/\partial z$ from Lesson 30 backward through the output projection to get $\partial L/\partial h$, then backward through the ReLU and the hidden projection to get $\partial L/\partial W_1, \partial L/\partial b_1$. Every step is either "multiply by a weight matrix's transpose" or "multiply elementwise by an activation's derivative" — this is all backpropagation ever is, no matter how many layers are stacked.

In [3]:
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def relu(z):
    return np.maximum(0, z)

def relu_deriv(z):
    return (z > 0).astype(np.float64)

def forward(X, W1, b1, W2, b2):
    z1 = X @ W1 + b1
    a1 = relu(z1)
    z2 = (a1 @ W2 + b2).ravel()
    p = sigmoid(z2)
    return p, (z1, a1, z2)

def backward(X, y, p, cache, W2):
    z1, a1, z2 = cache
    n = len(y)
    grad_z2 = ((p - y) / n).reshape(-1, 1)
    grad_W2 = a1.T @ grad_z2
    grad_b2 = grad_z2.sum(axis=0)
    grad_a1 = grad_z2 @ W2.T
    grad_z1 = grad_a1 * relu_deriv(z1)
    grad_W1 = X.T @ grad_z1
    grad_b1 = grad_z1.sum(axis=0)
    return grad_W1, grad_b1, grad_W2, grad_b2

Sanity check against PyTorch autograd

In [4]:
H = 4
init_rng = np.random.default_rng(8)
W1 = init_rng.normal(size=(2, H)) * 0.7
b1 = np.zeros(H)
W2 = init_rng.normal(size=(H, 1)) * 0.7
b2 = np.zeros(1)

p, cache = forward(X, W1, b1, W2, b2)
grad_W1, grad_b1, grad_W2, grad_b2 = backward(X, y, p, cache, W2)

X_t = torch.tensor(X)
y_t = torch.tensor(y)
W1_t = torch.tensor(W1, requires_grad=True)
b1_t = torch.tensor(b1, requires_grad=True)
W2_t = torch.tensor(W2, requires_grad=True)
b2_t = torch.tensor(b2, requires_grad=True)

z1_t = X_t @ W1_t + b1_t
a1_t = torch.relu(z1_t)
z2_t = (a1_t @ W2_t + b2_t).squeeze(-1)
loss_t = torch.nn.functional.binary_cross_entropy_with_logits(z2_t, y_t)
loss_t.backward()

print(f'W1 max diff: {np.abs(grad_W1 - W1_t.grad.numpy()).max():.2e}')
print(f'W2 max diff: {np.abs(grad_W2 - W2_t.grad.numpy()).max():.2e}')
print(f'b1 max diff: {np.abs(grad_b1 - b1_t.grad.numpy()).max():.2e}')
print(f'b2 max diff: {np.abs(grad_b2 - b2_t.grad.numpy()).max():.2e}')

W1 max diff: 6.94e-18
W2 max diff: 3.47e-18
b1 max diff: 2.78e-17
b2 max diff: 0.00e+00

Training

In [5]:
lr = 0.1
losses = []
for epoch in range(3000):
    p, cache = forward(X, W1, b1, W2, b2)
    eps = 1e-9
    losses.append(-np.mean(y * np.log(p + eps) + (1 - y) * np.log(1 - p + eps)))
    grad_W1, grad_b1, grad_W2, grad_b2 = backward(X, y, p, cache, W2)
    W1 -= lr * grad_W1; b1 -= lr * grad_b1
    W2 -= lr * grad_W2; b2 -= lr * grad_b2

final_pred = (p > 0.5).astype(float)
print(f'final accuracy: {(final_pred == y).mean():.1%}  (single neuron, Lesson 30, managed 50%)')

plt.plot(losses)
plt.xlabel('epoch'); plt.ylabel('loss')
plt.title('Training loss (2-layer MLP)')
plt.show()

final accuracy: 100.0%  (single neuron, Lesson 30, managed 50%)
No description has been provided for this image

What the hidden layer actually did

The output layer is just a linear projection (Lesson 30's neuron) — but it acts on $h$, the hidden layer's output, not on the original $x$. If the hidden layer did its job, the transformed points should already be much easier to separate with a straight line. Since $h$ lives in $\mathbb{R}^4$ here, we use PCA (Lesson 6) to visualize it in 2D.

In [6]:
_, (_, hidden, _) = forward(X, W1, b1, W2, b2)

hidden_centered = hidden - hidden.mean(axis=0)
cov = np.cov(hidden_centered.T)
eigvals, eigvecs = np.linalg.eigh(cov)
top2_directions = eigvecs[:, -2:]
hidden_2d = hidden_centered @ top2_directions

fig, axes = plt.subplots(1, 2, figsize=(9, 4))
axes[0].scatter(*X[y == 0].T, s=15, label='inner')
axes[0].scatter(*X[y == 1].T, s=15, label='outer')
axes[0].set_aspect('equal')
axes[0].set_title('Original input space')
axes[0].legend(fontsize=7)

axes[1].scatter(*hidden_2d[y == 0].T, s=15, label='inner')
axes[1].scatter(*hidden_2d[y == 1].T, s=15, label='outer')
axes[1].set_title('Hidden representation\n(top 2 PCA directions of a 4D space)')
axes[1].legend(fontsize=7)
plt.tight_layout()
plt.show()
No description has been provided for this image

Even this lossy 2D snapshot of the full 4D hidden space shows the two classes pulled apart into something close to linearly separable — a big improvement over the 74% ceiling that was the best any line could do in the original space (Lesson 29). The actual output neuron works in the full 4D hidden space, where it achieves 100% accuracy exactly. This is the entire mechanism of deep learning in miniature: each layer reshapes the space so that the next layer's job gets easier, until the final layer's job is trivial — a single linear projection.

Exercise

  1. Retrain with H = 2 (only 2 hidden units) instead of 4, trying a few different init_rng seeds. Can 2 hidden units ever reach 100% accuracy on this dataset, or does capacity this limited cap out lower? (Hint: think about how many straight cuts a ReLU layer with $H$ units can combine.)
  2. Replace relu/relu_deriv with tanh/its derivative ($1 - \tanh^2$) throughout, and retrain. Does it still solve the problem? Compare the resulting loss curve's shape to the ReLU version's.
  3. Remove the nonlinearity entirely (replace relu(z1) with just z1 in forward, and relu_deriv(z1) with an array of ones in backward). Confirm the network can no longer beat Lesson 30's ~50% ceiling, and explain algebraically why a linear hidden layer followed by a linear output layer is still just one big linear projection, no matter how many "layers" are stacked.
← Lesson 30: Neural Network Fundamentals — Learning the Projection
Lesson 32: Optimization →