Lesson 18: Color Spaces¶
Previous lessons converted color images to grayscale and moved on. This lesson looks at what that conversion actually does, and at the other ways to represent color besides RGB, each suited to a different task: HSV (separating color from brightness), YCbCr (separating luminance from chrominance, the basis of video compression from Lesson 17), and Lab (designed so distances match human perception).
import numpy as np
import cv2
import matplotlib.pyplot as plt
Grayscale is already a color-space choice¶
The simplest way to convert color to grayscale is to average the three channels:
$$Y = (R + G + B)/3$$
Although this kinda works, the human visual system is actually more sensitive to green than it is to the other colors. Therefore, it is better to weight the green more:
$$Y = (R + 2G + B)/4$$
An even better way is to leverage weights from actual experiments on human subjects, leading to the famous formula used in cv2.cvtColor(img, cv2.COLOR_BGR2GRAY):
$$Y = 0.299 R + 0.587 G + 0.114 B$$
These weights come from human luminance sensitivity: the eye is far more sensitive to green than to blue, so equal-intensity green and blue should not map to the same gray level, even though a naive channel average would treat them identically.
Why this formula is outdated (even though widely used, and still useful)¶
Two things are hiding in that formula that are worth knowing about:
- Old primaries. The weights $0.299, 0.587, 0.114$ come from the 1953 NTSC standard (Rec. 601), derived from the specific red/green/blue phosphors used in early color CRT televisions. Modern LED displays (and nearly everything else made since the early 2000s) use more saturated primaries, so the colorimetrically correct weights for a modern screen are actually $Y = 0.2126R + 0.7152G + 0.0722B$ (Rec. 709/sRGB): noticeably more green-weighted and less red-weighted.
cv2.cvtColor(..., COLOR_BGR2GRAY)still uses the old Rec. 601 weights, mainly for backward compatibility with decades of existing code. - Assumes linear light. The weighted-sum formula is only true luminance when $R$, $G$, $B$ are linear light values. But as we shall see in Lesson 25, stored pixel values are gamma-encoded, not linear.
cv2.cvtColorapplies the weights directly to the raw encoded 8-bit values, without decoding gamma first — fast, and close enough for most computer-vision purposes, but not the physically correct luminance a display-calibration or photometry application would need. (To see the problem, create a pure blue image with RGB=(0,0,255), then convert using OpenCV to get a value of 29 everywhere — which will incorrectly appear completely dark.)
weights_601 = np.array([0.114, 0.587, 0.299]) # BGR order, what cv2 actually uses (Rec. 601)
weights_709 = np.array([0.0722, 0.7152, 0.2126]) # BGR order, modern primaries (Rec. 709 / sRGB)
for name, bgr in [('pure red', (0, 0, 255)), ('pure green', (0, 255, 0))]:
bgr = np.array(bgr, dtype=np.float64)
y_601 = bgr @ weights_601
y_709 = bgr @ weights_709
print(f'{name:>10}: Rec. 601 (cv2) Y = {y_601:6.1f} Rec. 709 (modern) Y = {y_709:6.1f}')
pure_green = np.full((10, 10, 3), (0, 255, 0), dtype=np.uint8) # BGR
pure_blue = np.full((10, 10, 3), (255, 0, 0), dtype=np.uint8)
for name, patch in [('green', pure_green), ('blue', pure_blue)]:
weighted = cv2.cvtColor(patch, cv2.COLOR_BGR2GRAY)[0, 0]
naive = patch[0, 0].astype(np.float64).mean()
print(f'{name:>6}: cv2 (luminance-weighted) gray = {weighted:>3}, naive average = {naive:.0f}')
The naive average can't tell green and blue apart at all (both give 85); the perceptually weighted version correctly reports green as much brighter (150) than blue (29) at the same channel intensity.
RGB channels are highly correlated¶
In most real images, lighting/shading variation dominates: a surface gets brighter or darker as a whole, scaling all three channels together. That means R, G, and B end up strongly correlated with each other — not a very efficient or convenient way to separate "what color is this" from "how brightly lit is this."
rng = np.random.default_rng(0)
shading = rng.uniform(0.3, 1.0, (150, 150)) # a smoothly varying "lighting" field
shaded_img = np.stack([shading * 200, shading * 150, shading * 100], axis=-1).astype(np.uint8) # BGR
shaded_img = cv2.imread('../img/sheepdog.jpg')
r = shaded_img[:, :, 2].ravel().astype(np.float64)
g = shaded_img[:, :, 1].ravel().astype(np.float64)
print(f'correlation between R and G channels: {np.corrcoef(r, g)[0, 1]:.4f}')
img_h, img_w = shaded_img.shape[:2]
fig, axes = plt.subplots(1, 2, figsize=(4 * img_w / img_h + 4, 4),
gridspec_kw={'width_ratios': [img_w / img_h, 1]})
axes[0].imshow(cv2.cvtColor(shaded_img, cv2.COLOR_BGR2RGB))
axes[0].set_title('Image')
axes[0].axis('off')
axes[1].scatter(r, g, s=2, alpha=0.3)
axes[1].set_xlabel('R'); axes[1].set_ylabel('G')
axes[1].set_title('R and G are highly correlated')
axes[1].set_xlim(0, 255)
axes[1].set_ylim(0, 255)
axes[1].set_aspect('equal', adjustable='box')
plt.show()
HSV: separating color from brightness¶
HSV (Hue, Saturation, Value) reparametrizes color so that Hue captures which color (independent of how bright or washed-out it is), Saturation captures how vivid/pure it is, and Value captures brightness alone. This makes color-based segmentation dramatically more robust to lighting than thresholding directly in RGB.
size = 200
flat_img = np.zeros((size, size, 3), dtype=np.uint8)
cv2.circle(flat_img, (100, 100), 70, (0, 100, 255), -1) # an orange disk, BGR
yy, xx = np.mgrid[0:size, 0:size]
lighting_gradient = (0.25 + 0.9 * (xx / size)) # dark on the left, bright on the right
shaded_disk = np.clip(flat_img.astype(np.float64) * lighting_gradient[..., None], 0, 255).astype(np.uint8)
hsv = cv2.cvtColor(shaded_disk, cv2.COLOR_BGR2HSV)
hue = hsv[:, :, 0]
mask_hsv = (hue > 5) & (hue < 25) # hue range only
mask_rgb = ((shaded_disk[:, :, 2] > 150) & (shaded_disk[:, :, 1] > 50) &
(shaded_disk[:, :, 1] < 180) & (shaded_disk[:, :, 0] < 80)) # fixed RGB range
true_mask = (xx - 100)**2 + (yy - 100)**2 <= 70**2
def iou(a, b):
return (a & b).sum() / (a | b).sum()
print(f'IoU vs. true disk, HSV hue threshold: {iou(mask_hsv, true_mask):.3f}')
print(f'IoU vs. true disk, fixed RGB range: {iou(mask_rgb, true_mask):.3f}')
fig, axes = plt.subplots(1, 3, figsize=(9, 3.5))
axes[0].imshow(cv2.cvtColor(shaded_disk, cv2.COLOR_BGR2RGB))
axes[0].set_title('Shaded disk (left dark, right bright)')
axes[1].imshow(mask_hsv, cmap='gray')
axes[1].set_title('HSV hue threshold')
axes[2].imshow(mask_rgb, cmap='gray')
axes[2].set_title('Fixed RGB range')
for ax in axes:
ax.axis('off')
plt.tight_layout()
plt.show()
Hue thresholding recovers the disk perfectly regardless of the lighting gradient across it, while a fixed RGB range misses the darkened side entirely — exactly the brittleness that pure-RGB color segmentation runs into as soon as lighting isn't perfectly uniform.
YCbCr: luma and chroma, and why video compresses color more aggressively¶
YCbCr (used internally by JPEG and almost all video codecs) splits an image into luma ($Y$, roughly brightness) and two chroma channels ($C_b, C_r$, roughly "how blue" and "how red"). The human visual system resolves fine spatial detail in luma far better than in chroma — the biological basis for chroma subsampling (storing chroma at lower resolution than luma, e.g. video's common "4:2:0" format), which is one of the free compression wins used throughout Lesson 17's JPEG pipeline.
textured = np.zeros((200, 200, 3), dtype=np.uint8)
cv2.rectangle(textured, (20, 20), (90, 180), (255, 120, 30), -1)
cv2.circle(textured, (140, 100), 50, (30, 200, 255), -1)
gradient_x = np.mgrid[0:200, 0:200][1]
textured[:, :, 0] = np.clip(textured[:, :, 0].astype(int) + (gradient_x // 4) % 50, 0, 255)
textured = np.clip(textured.astype(np.float64) + rng.normal(0, 5, textured.shape), 0, 255).astype(np.uint8)
Y, Cr, Cb = cv2.split(cv2.cvtColor(textured, cv2.COLOR_BGR2YCrCb))
def subsample_then_upsample(channel, factor=8):
small = cv2.resize(channel, (channel.shape[1] // factor, channel.shape[0] // factor), interpolation=cv2.INTER_AREA)
return cv2.resize(small, (channel.shape[1], channel.shape[0]), interpolation=cv2.INTER_LINEAR)
chroma_degraded = cv2.cvtColor(cv2.merge([Y, subsample_then_upsample(Cr), subsample_then_upsample(Cb)]),
cv2.COLOR_YCrCb2BGR)
luma_degraded = cv2.cvtColor(cv2.merge([subsample_then_upsample(Y), Cr, Cb]), cv2.COLOR_YCrCb2BGR)
error_chroma = np.abs(chroma_degraded.astype(int) - textured.astype(int)).mean()
error_luma = np.abs(luma_degraded.astype(int) - textured.astype(int)).mean()
print(f'mean abs error, chroma degraded 8x: {error_chroma:.2f}')
print(f'mean abs error, luma degraded 8x: {error_luma:.2f}')
fig, axes = plt.subplots(1, 3, figsize=(9, 3.5))
for ax, im, title in zip(axes, [textured, chroma_degraded, luma_degraded],
['Original', 'Chroma subsampled 8x\n(less visible loss)', 'Luma subsampled 8x\n(more visible loss)']):
ax.imshow(cv2.cvtColor(im, cv2.COLOR_BGR2RGB))
ax.set_title(title, fontsize=9)
ax.axis('off')
plt.tight_layout()
plt.show()
Discarding the same amount of resolution costs noticeably less error in chroma than in luma — the same detail loss is simply less objectionable when it happens to color rather than brightness, which is exactly the tradeoff chroma subsampling exploits.
L*a*b*: a perceptually uniform space¶
RGB's Euclidean distance is a poor stand-in for how different two colors actually look: equal RGB distances can correspond to wildly different perceived differences, depending on where in color space they fall. CIELAB was explicitly designed so that Euclidean distance between two L*a*b* colors approximates perceptual difference much more consistently — useful for color-based quality metrics, clustering, and matching.
target_rgb_distance = 30.0
n_samples = 2000
c1 = rng.uniform(0, 255, (n_samples, 3))
direction = rng.normal(size=(n_samples, 3))
direction /= np.linalg.norm(direction, axis=1, keepdims=True)
c2 = c1 + direction * target_rgb_distance
valid = np.all((c2 >= 0) & (c2 <= 255), axis=1)
c1, c2 = c1[valid].astype(np.uint8), c2[valid].astype(np.uint8)
lab1 = cv2.cvtColor(c1.reshape(1, -1, 3), cv2.COLOR_BGR2LAB).reshape(-1, 3).astype(np.float64)
lab2 = cv2.cvtColor(c2.reshape(1, -1, 3), cv2.COLOR_BGR2LAB).reshape(-1, 3).astype(np.float64)
lab_distances = np.linalg.norm(lab1 - lab2, axis=1)
print(f'RGB distance held fixed at exactly {target_rgb_distance}')
print(f'resulting L*a*b* distances: min={lab_distances.min():.1f}, '
f'median={np.median(lab_distances):.1f}, max={lab_distances.max():.1f}')
plt.hist(lab_distances, bins=40)
plt.xlabel('L*a*b* distance')
plt.ylabel('count')
plt.title(f'Perceptual (L*a*b*) distance for {n_samples} color pairs,\nall with the SAME RGB distance ({target_rgb_distance})')
plt.show()
Every one of these pairs is exactly the same distance apart in RGB — yet the corresponding perceptual (L*a*b*) difference ranges from barely noticeable to more than 15x that, depending purely on where in color space the pair sits. Any algorithm using raw RGB distance as a proxy for "how similar do these colors look" (nearest-neighbor color matching, k-means color clustering, background-subtraction thresholds) inherits this same distortion; switching to Lab distance is a simple, standard fix.
Exercise¶
- Repeat the HSV vs. RGB segmentation demo with a shading gradient that also shifts hue slightly (e.g. blend toward a different color at one edge, not just changing brightness). Does hue-only thresholding still work as well?
- Increase the chroma/luma subsampling factor from 8 to 20. Does the qualitative story (chroma loss less visible than luma loss) still hold, or does it eventually break down?
- Find, by inspecting
c1/c2in the Lab experiment, one example pair with a very small Lab distance and one with a very large L*a*b* distance (despite both having the same RGB distance). Display the four colors as swatches and describe, in your own words, why the low-Lab-distance pair looks more similar.