Week 7 · Hands-on foundations

Multilayer Perceptrons and Activations

What changes when a linear model gains hidden layers and nonlinear activations?

The current Week 7 material covers theory, from-scratch MLPs, concise implementations, deeper networks, activations, regression, and project applications in one large notebook. The two clean 2025 notebooks provide a better teaching spine: first understand hidden layers and activations, then implement one compact network.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Explain why stacked linear layers remain linear without an activation.
Compare ReLU, sigmoid, and tanh output ranges.
Build an MLP with `nn.Sequential`.
Relate width and depth to parameter count and overfitting risk.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Hold out validation data and compare ReLU, tanh, and a linear model using identical seeds and training budgets.

Core ideas

Three concepts to keep

Hidden representation

A hidden layer transforms raw features into learned intermediate features. These representations are useful only if the end-to-end task supplies evidence that they improve held-out behavior.

Nonlinearity

Without a nonlinear activation, several affine layers collapse into one affine transformation. Activations let the network bend and combine decision regions.

Width and depth

Width controls units per layer; depth controls stacked transformations. Both increase capacity and optimization choices, so a simple baseline remains important.

Mathematical intuition

One relationship worth keeping

\[\mathbf{h}=\phi(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1),\qquad \hat{\mathbf{y}}=\mathbf{W}_2\mathbf{h}+\mathbf{b}_2\]

The activation φ is the key difference from one linear transformation. Its position and range shape both representation and gradient flow.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Input -1Network depth 3Bias 0

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Keep the Week 6 split and metric contract unchanged.
2. Fit the linear classifier as the baseline.
3. Add one small hidden layer and one activation.
4. Compare validation behavior and parameter count.
5. Increase depth or width only when a diagnosed limitation justifies it.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Compare activations

import math
x = -1.0
relu = max(0, x)
sigmoid = 1/(1+math.exp(-x))
tanh = math.tanh(x)
print(round(relu,3), round(sigmoid,3), round(tanh,3))

output

Predict the result, then click Run.

Count MLP parameters

inputs, hidden, outputs = 4, 8, 3
params = inputs*hidden + hidden + hidden*outputs + outputs
print(params)

output

Predict the result, then click Run.

Show why two linear layers collapse

w1, b1, w2, b2, x = 2, 1, 3, -2, 4
stacked = w2*(w1*x+b1)+b2
collapsed = (w2*w1)*x + (w2*b1+b2)
print(stacked, collapsed)

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

No activation

What goes wrong: Several layers still represent one linear map.

Check: Inspect the model sequence.

Inactive ReLU units

What goes wrong: A unit receives negative inputs and outputs zero throughout training.

Check: Inspect activation distributions or try a smaller rate/better initialization.

Saturated sigmoid

What goes wrong: Large inputs produce tiny gradients.

Check: Standardize inputs and compare activation ranges.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. What changes when a linear model gains hidden layers and nonlinear activations?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. The two clean 2025 notebooks provide a better teaching spine: first understand hidden layers and activations, then implement one compact network.

2. What does hidden representation mean here?

Standard answer: A hidden layer transforms raw features into learned intermediate features. These representations are useful only if the end-to-end task supplies evidence that they improve held-out behavior.

3. Why is nonlinearity useful?

Standard answer: Without a nonlinear activation, several affine layers collapse into one affine transformation. Activations let the network bend and combine decision regions.

4. How should you interpret width and depth?

Standard answer: Width controls units per layer; depth controls stacked transformations. Both increase capacity and optimization choices, so a simple baseline remains important.

5. What does the main equation clarify—and what does it not prove?

Standard answer: The activation φ is the key difference from one linear transformation. Its position and range shape both representation and gradient flow. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Keep the Week 6 split and metric contract unchanged.

8. How can you detect the failure called “No activation”?

Standard answer: Inspect the model sequence.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 8 turns MLP components into maintainable models: inspecting parameters, saving state, debugging shapes, and preparing a small project.

Terminology

Glossary

Perceptron: Linear unit followed by a decision or activation.
MLP: Feedforward network with one or more hidden layers.
Hidden layer: Intermediate learned transformation between input and output.
Activation: Nonlinear function applied to a layer output.
ReLU: Activation returning max(0, x).
Sigmoid: Activation mapping values to the interval zero to one.
Tanh: Activation mapping values to minus one to one.
Width: Number of units in a layer.
Depth: Number of stacked learned layers.
Universal approximation: Capacity result showing a sufficiently large network can approximate broad function classes.

Go further

Key references

Next: Week 8 turns MLP components into maintainable models: inspecting parameters, saving state, debugging shapes, and preparing a small project.