By the end
- Express linear regression as `nn.Linear`.
- Explain predictions, residuals, and mean squared error.
- Use batches to fit a weight and bias.
- Compare learned and hand-specified parameters.
Week 3 · Hands-on foundations
Linear Regression as a Neural Network
How can a familiar linear model become a complete learning workflow?
The long Week 3–4 regression notebook connects analytic regression, minibatch gradient descent, object-oriented code, real data, and nonlinear models. This chapter keeps the conceptual bridge: a linear equation is also a one-layer neural network. Students fit it first with compact PyTorch components and inspect residuals before adding complexity.
This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.
Low floor
Use the explorer and three short Python examples before changing a longer model.
High ceiling
Add a second feature and explain why the visualization becomes harder even though `nn.Linear` changes very little.
Core ideas
Three concepts to keep
Model
A linear regressor combines each feature with a learned weight and adds a bias. The output is continuous, so the model represents a plane or hyperplane rather than a class boundary.
Residual and squared loss
A residual is observed minus predicted. Squaring makes large errors matter more and removes cancellation between positive and negative residuals.
Batch learning
A batch estimates the average gradient from several observations. Minibatches balance noisy single-example updates against expensive full-dataset updates.
Mathematical intuition
One relationship worth keeping
\[\hat y_i=\mathbf{x}_i^\top\mathbf{w}+b,\qquad \mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat y_i)^2\]
The model maps features to a prediction; MSE maps all residuals to one optimization signal. Neither equation alone proves the relationship will generalize.
Stretch: read the symbols slowly
Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.
Interactive explorer
Change one assumption at a time
Predict the direction of change before moving a control.
Current evidence—
Interpretation—
Change a control to inspect the relationship.
Hands-on path
A reusable five-step routine
- 1. Plot or tabulate the target against each important feature.
- 2. Split before fitting preprocessing or model parameters.
- 3. Build the smallest linear baseline.
- 4. Train while recording loss, not just the final number.
- 5. Inspect residuals and held-out error before trying a nonlinear model.
Minimal Python
Predict, run, and explain
Each example is independent. Explain its output before copying it into a larger workflow.
Make linear predictions
x = [0., 1., 2.]
w, b = 1.5, 0.5
print([w * value + b for value in x])output
Predict the result, then click Run.
Calculate mean squared error
y = [1., 2., 4.]
pred = [0.5, 2., 3.5]
mse = sum((a-b)**2 for a,b in zip(y,pred)) / len(y)
print(round(mse, 3))output
Predict the result, then click Run.
Count a linear layer's parameters
features, outputs = 4, 1
weights = features * outputs
biases = outputs
print(weights + biases)output
Predict the result, then click Run.
Failure checks
What can look correct while being wrong?
Scale imbalance
What goes wrong: One feature dominates gradient updates because its numeric range is much larger.
Check: Compare feature ranges and standardize from training data.
Target leakage
What goes wrong: A feature contains information created after the outcome.
Check: Write the prediction time and inspect feature availability.
Average-only evaluation
What goes wrong: Low MSE hides systematic failure for part of the range.
Check: Plot residuals against predictions and important slices.
Check your understanding
Ten questions with standard answers
Answer in your own words before opening each panel.
1. How can a familiar linear model become a complete learning workflow?
Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. Students fit it first with compact PyTorch components and inspect residuals before adding complexity.
2. What does model mean here?
Standard answer: A linear regressor combines each feature with a learned weight and adds a bias. The output is continuous, so the model represents a plane or hyperplane rather than a class boundary.
3. Why is residual and squared loss useful?
Standard answer: A residual is observed minus predicted. Squaring makes large errors matter more and removes cancellation between positive and negative residuals.
4. How should you interpret batch learning?
Standard answer: A batch estimates the average gradient from several observations. Minibatches balance noisy single-example updates against expensive full-dataset updates.
5. What does the main equation clarify—and what does it not prove?
Standard answer: The model maps features to a prediction; MSE maps all residuals to one optimization signal. Neither equation alone proves the relationship will generalize. It does not by itself prove useful behavior on unseen intended-use cases.
6. What should change when you move the explorer controls?
Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.
7. What is the first hands-on check you should perform?
Standard answer: Plot or tabulate the target against each important feature.
8. How can you detect the failure called “Scale imbalance”?
Standard answer: Compare feature ranges and standardize from training data.
9. What evidence should be recorded before making a claim?
Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.
10. How does this week prepare the next topic?
Standard answer: Week 4 keeps the regression task but focuses on the reusable training loop, validation, and the point at which added capacity stops helping.
Terminology
Glossary
- Regression
- Prediction of a continuous numeric target.
- Feature
- Input variable used by a model.
- Target
- Quantity the model is trained to predict.
- Weight
- Learned coefficient multiplying a feature.
- Bias
- Learned intercept added to a linear transformation.
- Prediction
- Model output for a given input.
- Residual
- Observed target minus prediction.
- MSE
- Mean of squared residuals.
- Minibatch
- Small subset used for one parameter update.
- Baseline
- Simple reference model that a complex model must improve upon.
Go further
Key references
- PyTorch linear regression tutorial
- Dive into Deep Learning: Linear Regression
- scikit-learn regression metrics
Next: Week 4 keeps the regression task but focuses on the reusable training loop, validation, and the point at which added capacity stops helping.