Week 4 · Hands-on foundations

Training Loops and Generalization

When does a falling training loss become a useful model?

The 2025 refined regression notebooks add early stopping, learning curves, nonlinear comparisons, and cross-validation. They are valuable but dense for one class. This week isolates the reusable loop—split, train, validate, stop, and compare—so later architectures can reuse the same evidence discipline.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Write the five stages of a PyTorch training loop.
Keep training, validation, and test roles separate.
Read learning curves for underfitting and overfitting.
Use early stopping without tuning on the test set.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Repeat with three seeds and compare the best validation epoch and loss.

Core ideas

Three concepts to keep

Optimization loop

Each iteration calculates predictions and loss, clears old gradients, runs backward, and updates parameters. Evaluation mode omits gradient recording and parameter updates.

Generalization

A model generalizes when its behavior transfers to unseen examples from the intended use distribution. Training fit is necessary but not sufficient evidence.

Early stopping

Validation performance can identify when additional fitting stops transferring. The test set remains untouched until the model and stopping rule are fixed.

Mathematical intuition

One relationship worth keeping

\[\mathrm{gap}=L_{\mathrm{validation}}-L_{\mathrm{training}}\]

A growing gap can indicate overfitting, but both losses must come from a valid split. A small gap between two bad losses still represents underfitting.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Learning rate 0.2Epochs 20Capacity 4

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Freeze a reproducible split and preprocessing rule.
2. Train only on the training portion.
3. Record training and validation loss at the same checkpoints.
4. Choose stopping and hyperparameters from validation evidence.
5. Evaluate the frozen choice once on test data and document variability.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Read a learning gap

train_loss = 0.18
validation_loss = 0.31
print(round(validation_loss - train_loss, 2))

output

Predict the result, then click Run.

Select an early-stopping epoch

validation = [0.62, 0.44, 0.36, 0.35, 0.39]
best_epoch = min(range(len(validation)), key=validation.__getitem__) + 1
print(best_epoch)

output

Predict the result, then click Run.

Summarize repeated runs

scores = [0.78, 0.81, 0.79, 0.80]
print(round(sum(scores)/len(scores), 3), round(max(scores)-min(scores), 3))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Test-set tuning

What goes wrong: Repeated test checks influence model choices.

Check: Use validation for iteration and reserve the test set.

Mode confusion

What goes wrong: Dropout or batch normalization behaves differently during evaluation.

Check: Use `model.train()` and `model.eval()` deliberately.

Single lucky seed

What goes wrong: A claim depends on one random initialization or split.

Check: Repeat the small experiment with several seeds.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. When does a falling training loss become a useful model?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. This week isolates the reusable loop—split, train, validate, stop, and compare—so later architectures can reuse the same evidence discipline.

2. What does optimization loop mean here?

Standard answer: Each iteration calculates predictions and loss, clears old gradients, runs backward, and updates parameters. Evaluation mode omits gradient recording and parameter updates.

3. Why is generalization useful?

Standard answer: A model generalizes when its behavior transfers to unseen examples from the intended use distribution. Training fit is necessary but not sufficient evidence.

4. How should you interpret early stopping?

Standard answer: Validation performance can identify when additional fitting stops transferring. The test set remains untouched until the model and stopping rule are fixed.

5. What does the main equation clarify—and what does it not prove?

Standard answer: A growing gap can indicate overfitting, but both losses must come from a valid split. A small gap between two bad losses still represents underfitting. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Freeze a reproducible split and preprocessing rule.

8. How can you detect the failure called “Test-set tuning”?

Standard answer: Use validation for iteration and reserve the test set.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Weeks 5 and 6 reuse the same split/train/validate pattern for classification, adding probabilities, thresholds, and class-specific errors.

Terminology

Glossary

Epoch: One pass through the training data.
Training set: Data used to update parameters.
Validation set: Held-out data used for model choices.
Test set: Final held-out data used after choices are frozen.
Generalization: Transfer of learned behavior to unseen intended-use cases.
Underfitting: Failure to capture important structure even in training data.
Overfitting: Training-specific fit that does not transfer.
Early stopping: Ending training based on validation evidence.
Seed: Value controlling a reproducible random sequence.
Learning curve: Metric tracked across training time or data amount.

Go further

Key references

Next: Weeks 5 and 6 reuse the same split/train/validate pattern for classification, adding probabilities, thresholds, and class-specific errors.