Week 12
Image Analysis and Audio Signal Processing
Images and sounds are data. This chapter shows how pixels, filters, waves, and spectrograms become numerical representations that computers can analyze.
Pixels
Filters
Signals
Learning Goals
By the End of This Chapter
By the end, you should be able to describe images as pixel arrays, explain smoothing and edge filters, describe amplitude/frequency/phase, and interpret a spectrogram as time-frequency data.
How to Use This Page
Try each explorer first, then run the related script. The explorers build intuition; the code shows how the notebook implements the same ideas.
Source Materials
The notebook supplies the Python examples. The slides add the conceptual sequence: pixels and resolution, convolution, edge filters, sound waves, sampling, spectra, and spectrograms.
Part 1
Digital Images
An image is a grid of pixels. A grayscale image stores one intensity value per pixel, while an RGB image stores three values: red, green, and blue.
image = height x width x channels
Image as a function
The slides describe a grayscale image as a map f(x, y) from pixel position to intensity. In an 8-bit image, that intensity often ranges from 0 for black to 255 for white.
Pixel Grid Explorer
Each cell is one pixel. Changing the channel view shows why an RGB image has three values per pixel, while grayscale has one intensity value.
Load image and convert to grayscale
from skimage import data, color
image_rgb = data.astronaut()
image = color.rgb2gray(image_rgb)
print("RGB image shape:", image_rgb.shape)
print("Grayscale image shape:", image.shape)
print("Grayscale value range: 0.0 to 1.0")Part 2
Smoothing and Convolution
Image filtering modifies pixel values by looking at nearby pixels. A smoothing filter replaces each pixel with a local average, which reduces noise but also blurs edges.
Key point
A convolution kernel is a small matrix of weights. The kernel slides over the image, multiplies nearby pixel values by those weights, and sums the result.
Average filter
The notebook uses a 3x3 average filter. Every weight is 1/9, so each output pixel is the mean of its 3x3 neighborhood.
Convolution Explorer
The left grid is a small grayscale image patch. The blue square marks the 3x3 neighborhood used to compute the output pixel. Choose a kernel to see how the weighted sum changes.
Create a 3x3 average filter
import numpy as np
average_filter = np.ones((3, 3)) / 9.0
print(average_filter)Apply smoothing convolution
from scipy.ndimage import convolve
filtered_image = convolve(image, average_filter)
print("Filtered image shape:", filtered_image.shape)
print("Effect: local averaging reduces noise and softens edges.")Part 3
Edge Detection
Edges are sudden changes in intensity. Sobel filters estimate gradients, while Canny combines smoothing, gradient detection, edge thinning, and thresholding.
gradient magnitude = sqrt(Gx^2 + Gy^2)
Sobel and Canny
Sobel emphasizes local intensity changes. Canny adds Gaussian smoothing controlled by sigma, so a larger sigma keeps fewer, smoother edges.
Edge Detector Explorer
Sobel uses gradients and does not use sigma. Canny first smooths the image, so sigma changes which edges remain.
Sobel edge detection
from skimage.filters import sobel
edge_sobel = sobel(image)
print("Sobel edge image shape:", edge_sobel.shape)
print("Output meaning: gradient magnitude at each pixel.")Canny edge detection
from skimage.feature import canny
edge_canny = canny(image, sigma=1.0)
print("Canny edge mask shape:", edge_canny.shape)
print("Data type: bool")
print("sigma: 1.0")Part 4
Sound as a Signal
Sound is a wave. In data analysis, a digital audio file becomes a sequence of amplitude samples measured at a fixed sampling rate.
s(t) = A sin(2 pi f t + phase)
- Amplitude controls loudness.
- Frequency controls pitch.
- Period is the time for one cycle.
- Phase shifts where the wave starts.
Waveform Explorer
Load Brahms audio with librosa
import librosa
audio_path = librosa.example("brahms")
audio_data, sample_rate = librosa.load(audio_path, duration=45)
print(f"Audio data shape: {audio_data.shape}, Sample rate: {sample_rate} Hz")Create audio playback object
from IPython.display import Audio
player = Audio(audio_data, rate=sample_rate)
print("Audio object prepared for playback at", sample_rate, "Hz")Plot time-domain waveform
import matplotlib.pyplot as plt
plt.plot(audio_data)
plt.title("Time-Domain Waveform")
plt.xlabel("Samples")
plt.ylabel("Amplitude")
plt.show()
print("Waveform samples:", len(audio_data))Part 5
Spectrograms
A spectrogram shows how frequency content changes over time. It is often the bridge between raw audio and machine learning models.
How to read it
The x-axis is time, the y-axis is frequency, and color represents amplitude in decibels. Each vertical slice is a short-time frequency spectrum.
STFT process
The Short-Time Fourier Transform cuts audio into windows, computes a Fourier transform for each window, then stacks those spectra over time.
Spectrum and Spectrogram Explorer
The signal combines 50 Hz, 120 Hz, and 300 Hz components from the slides. Change component strength and energy pattern to see the spectrum bars and spectrogram heatmap respond.
Generate a spectrogram
import librosa.display
D = librosa.stft(audio_data)
D_db = librosa.amplitude_to_db(abs(D))
librosa.display.specshow(D_db, sr=sample_rate)
plt.colorbar(format="%+2.0f dB")
plt.title("Spectrogram")
plt.show()
print("STFT matrix shape:", D.shape)
print("Spectrogram dB shape:", D_db.shape)Practice
Practice Questions
Use these questions to check your understanding and connect the examples to real image and audio data.
Channels
What information is lost when an RGB image is converted to grayscale, and why might that still be useful?
Filtering
Why does an average filter reduce noise but make edges less sharp?
Edges
How does increasing Canny sigma change the kinds of edges that remain?
Spectrograms
Why might a spectrogram be easier for a model to use than raw audio samples?
Reference
Key Terms
| Term | Meaning | Section |
|---|---|---|
| Pixel | The smallest addressable unit in a digital image. | Digital images |
| Channel | One layer of image values, such as red, green, blue, or grayscale intensity. | Digital images |
| Convolution | A weighted local operation using a sliding kernel. | Smoothing |
| Sobel | An edge detector based on horizontal and vertical gradients. | Edges |
| Canny | A multi-step edge detector with smoothing, gradients, thinning, and thresholding. | Edges |
| Sampling rate | The number of audio samples measured per second. | Sound |
| Frequency | The number of wave cycles per second, measured in Hertz. | Sound |
| Spectrogram | A time-frequency representation of audio intensity. | Spectrograms |