Lesson 27: Camera Calibration¶
Lesson 26 assumed the camera intrinsics $K$ were already known. This lesson shows how to actually get them: camera calibration, typically done by showing the camera a known flat pattern (a checkerboard) from several different angles. The classic approach (Zhang, 2000) works by extracting a homography between the checkerboard plane and each image — a direct callback to Lesson 23 — and combining constraints from several such homographies to pin down $K$.
import numpy as np
import cv2
import matplotlib.pyplot as plt
What calibration recovers¶
Two separate things:
- Intrinsics $K = \begin{bmatrix}f_x & s & c_x\\0 & f_y & c_y\\0&0&1\end{bmatrix}$: focal lengths, principal point, and skew — the ideal pinhole part of the camera model used throughout Lessons 26 and 28.
- Distortion coefficients $(k_1, k_2, p_1, p_2, k_3, \dots)$: real lenses aren't perfect pinholes. Radial distortion ($k_1, k_2, k_3$) bows straight lines into curves (barrel or pincushion, depending on sign); tangential distortion ($p_1, p_2$) accounts for the lens not being perfectly parallel to the sensor.
Every equation in Lessons 26 and 28 secretly assumed distortion was already removed — calibration is the step that makes that assumption true.
A synthetic calibration rig¶
Since we don't have a physical camera and checkerboard handy, we build one entirely synthetically: define a ground-truth $K$ and distortion, define a flat checkerboard's 3D corner positions, and generate several "photos" of it from different poses by projecting the corners with cv2.projectPoints (which applies distortion exactly as a real lens would). This gives us the same kind of input cv2.calibrateCamera expects from real detected checkerboard corners — with the enormous benefit of also knowing the ground truth to check against.
rng = np.random.default_rng(0)
cols, rows = 9, 6 # internal corners of a 10x7-square checkerboard
corners_3d = np.zeros((rows * cols, 3), dtype=np.float64)
corners_3d[:, :2] = np.mgrid[0:cols, 0:rows].T.reshape(-1, 2)
corners_3d[:, :2] -= corners_3d[:, :2].mean(axis=0) # center the board on its own origin
K_true = np.array([[800, 0, 320], [0, 800, 240], [0, 0, 1]], dtype=np.float64)
dist_true = np.array([-0.3, 0.1, 0.001, -0.0005, 0.02]) # k1, k2, p1, p2, k3
image_size = (640, 480)
object_points, image_points = [], []
for _ in range(40):
rvec = rng.uniform(-0.4, 0.4, 3) # a somewhat random tilt
tvec = np.array([rng.uniform(-0.5, 0.5), rng.uniform(-0.4, 0.4), rng.uniform(9, 13)])
projected, _ = cv2.projectPoints(corners_3d, rvec, tvec, K_true, dist_true)
projected = projected.reshape(-1, 2)
if np.all(projected >= 5) and np.all(projected[:, 0] < image_size[0] - 5) and np.all(projected[:, 1] < image_size[1] - 5):
object_points.append(corners_3d.astype(np.float32))
image_points.append(projected.astype(np.float32))
print(f'{len(object_points)} usable synthetic views (out of 40 attempted; some fell outside the frame)')
What the calibration target actually looks like¶
corners_3d and image_points above are just numbers so far. To make the target concrete: a real checkerboard is a flat pattern of alternating squares, and since it's flat, the mapping from the board's own 2D surface coordinates to a photo of it is exactly a homography — the same one Zhang's method extracts. We can use that fact in reverse: cv2.findHomography between the board's flat-pattern pixel coordinates and two of the synthetic image_points views, then warp a picture of the actual checkerboard through it, to see what those two synthetic "photos" would really look like.
sq = 60 # pixels per square in the flat target image
board = np.zeros(((rows + 1) * sq, (cols + 1) * sq), dtype=np.uint8)
for r in range(rows + 1):
for c in range(cols + 1):
if (r + c) % 2 == 0:
board[r * sq:(r + 1) * sq, c * sq:(c + 1) * sq] = 255
board_bgr = cv2.cvtColor(board, cv2.COLOR_GRAY2BGR)
# the internal corners' locations in the flat target's own pixel coordinates,
# in the same order as corners_3d
board_pts = np.array([[(c + 1) * sq, (r + 1) * sq] for r in range(rows) for c in range(cols)],
dtype=np.float32)
fig, axes = plt.subplots(1, 3, figsize=(12, 4))
axes[0].imshow(board, cmap='gray')
axes[0].set_title('Flat checkerboard target')
axes[0].axis('off')
for i, ax in zip([0, 1], axes[1:]):
H, _ = cv2.findHomography(board_pts, image_points[i])
photo = cv2.warpPerspective(board_bgr, H, image_size, borderValue=(128, 128, 128))
cv2.drawChessboardCorners(photo, (cols, rows), image_points[i], True)
ax.imshow(cv2.cvtColor(photo, cv2.COLOR_BGR2RGB))
ax.set_title(f'Synthetic photo {i}')
ax.axis('off')
plt.tight_layout()
plt.show()
Running calibration¶
cv2.calibrateCamera takes the (known) 3D corner positions and their (detected/here, projected) 2D image positions across all views, and jointly solves for $K$, distortion, and each view's own pose.
rms_error, K_estimated, dist_estimated, rvecs, tvecs = cv2.calibrateCamera(
object_points, image_points, image_size, None, None)
print(f'RMS reprojection error: {rms_error:.2e} pixels\n')
print('K (true):\n', K_true)
print('K (estimated):\n', np.round(K_estimated, 3))
print()
print('distortion (true): ', dist_true)
print('distortion (estimated):', np.round(dist_estimated.ravel(), 5))
With clean (noiseless) synthetic correspondences, calibration recovers the ground truth essentially exactly. Real calibration never sees noise-free data — corner detection on a real photo introduces sub-pixel jitter — which is exactly why in practice you'd use as many views, from as wide a variety of angles, as practical: more (and more diverse) constraints average out that noise.
Straight lines, bent by a lens¶
To see distortion directly (rather than just as numbers), apply the same distortion model used above to points sampled along perfectly straight lines, in normalized (pre-$K$) camera coordinates:
$$x_d = x(1 + k_1r^2 + k_2r^4 + k_3r^6) + 2p_1xy + p_2(r^2+2x^2)$$ $$y_d = y(1 + k_1r^2 + k_2r^4 + k_3r^6) + p_1(r^2+2y^2) + 2p_2xy$$
where $r^2 = x^2+y^2$. This is exactly the model cv2.projectPoints applied internally above.
def distort_normalized(xy, dist):
k1, k2, p1, p2, k3 = dist
x, y = xy[:, 0], xy[:, 1]
r2 = x**2 + y**2
radial = 1 + k1 * r2 + k2 * r2**2 + k3 * r2**3
xd = x * radial + 2 * p1 * x * y + p2 * (r2 + 2 * x**2)
yd = y * radial + p1 * (r2 + 2 * y**2) + 2 * p2 * x * y
return np.stack([xd, yd], axis=1)
def make_grid_lines():
lines = []
for xv in np.linspace(-0.38, 0.38, 9):
ys = np.linspace(-0.28, 0.28, 50)
lines.append(np.stack([np.full_like(ys, xv), ys], axis=1))
for yv in np.linspace(-0.28, 0.28, 7):
xs = np.linspace(-0.38, 0.38, 50)
lines.append(np.stack([xs, np.full_like(xs, yv)], axis=1))
return lines
def draw_lines(lines_px, height=480, width=640):
img = np.full((height, width, 3), 255, dtype=np.uint8)
for line in lines_px:
for i in range(len(line) - 1):
cv2.line(img, tuple(line[i].astype(int)), tuple(line[i + 1].astype(int)), (0, 0, 0), 1)
return img
grid_lines_normalized = make_grid_lines()
straight_px = [line @ K_true[:2, :2].T + K_true[:2, 2] for line in grid_lines_normalized]
distorted_px = [distort_normalized(line, dist_true) @ K_true[:2, :2].T + K_true[:2, 2]
for line in grid_lines_normalized]
fig, axes = plt.subplots(1, 2, figsize=(9, 4))
axes[0].imshow(draw_lines(straight_px))
axes[0].set_title('Ideal pinhole (no distortion)')
axes[1].imshow(draw_lines(distorted_px))
axes[1].set_title(f'Through the lens\n(k1={dist_true[0]}, k2={dist_true[1]})')
for ax in axes:
ax.axis('off')
plt.tight_layout()
plt.show()
Undoing distortion with the estimated parameters¶
cv2.undistortPoints inverts the distortion model, converting distorted pixel coordinates back to normalized undistorted coordinates. Applying it to the bent grid lines above, using our estimated (not the true) calibration, should straighten them back out.
undistorted_normalized = [
cv2.undistortPoints(px.reshape(-1, 1, 2).astype(np.float64), K_estimated, dist_estimated).reshape(-1, 2)
for px in distorted_px
]
undistorted_px = [line @ K_estimated[:2, :2].T + K_estimated[:2, 2] for line in undistorted_normalized]
fig, axes = plt.subplots(1, 2, figsize=(9, 4))
axes[0].imshow(draw_lines(distorted_px))
axes[0].set_title('Distorted (as captured)')
axes[1].imshow(draw_lines(undistorted_px))
axes[1].set_title('Undistorted using estimated K, dist')
for ax in axes:
ax.axis('off')
plt.tight_layout()
plt.show()
original_normalized = np.concatenate(grid_lines_normalized)
recovered_normalized = np.concatenate(undistorted_normalized)
print(f'max recovery error (normalized coords): {np.abs(original_normalized - recovered_normalized).max():.2e}')
The lines are straight again, and the recovered points match the original ideal grid to within floating-point precision — the whole point of calibrating a real camera is to be able to do exactly this correction on real photos before handing them to any of the geometric machinery from Lessons 26 and 28, all of which implicitly assumes an ideal, distortion-free pinhole.
Exercise¶
- Add pixel noise (e.g.
rng.normal(0, 0.5, projected.shape)) to the synthetic corner detections before calibrating. How much does the RMS reprojection error grow, and how doesK_estimateddrift fromK_true? - Reduce the number of usable calibration views from ~10-15 down to just 3. Does the estimate noticeably degrade, especially for the distortion coefficients (which need corners spread widely across the frame, including near the edges, to be well constrained)?
- Set
dist_true = np.zeros(5)(a perfect pinhole, no distortion) and rerun the whole notebook. Confirm the "distorted" and "ideal" grids become identical, and that calibration still recoversK_truecorrectly even with zero distortion to estimate.