Week 3 Building and Evaluating AI Systems
Generalization and Experimental Discipline
Low training loss is evidence about observed training cases. Generalization requires a defensible future population, a split that imitates that future, repeated experiments, and explicit tests for leakage and shift.
Core question: How do we know a model learned something that will transfer?
By the end
- Distinguish training, validation, test, and deployment populations.
- Select random, grouped, or temporal splits from the intended use.
- Interpret regularization and early stopping as controlled bias–variance decisions.
- Launch PBL1 with a reproducible baseline and experiment registry.
Course progression
Build one evidence chain
This week uses the course-wide sequence: frame the use, establish a baseline, build or compare, evaluate failure, document boundaries, and decide.
framebaselinebuildevaluatedocumentdecidemonitor
Connection from the course
Week 4 asks what architectural assumptions may improve transfer for images and sequences—and when those assumptions are unnecessary.
Essential concepts
Understand the parts before combining them
The core lane focuses on transferable judgment. Optional formal or engineering depth belongs in the companion notebook's stretch lane.
Generalization gap
The difference between training and held-out behavior can signal overfitting, but a small gap is meaningful only when the held-out sample represents the intended future.
Split as a causal boundary
Random splits assume exchangeable cases. Grouped splits prevent identity leakage; temporal splits prevent learning from the future.
Regularization
Weight decay, dropout, early stopping, augmentation, and smaller models restrict or disturb fitting. Their value is judged on valid held-out evidence.
Mathematical intuition
One relationship worth keeping
\[\widehat{R}_{\text{test}}(f)=\frac{1}{n_{\text{test}}}\sum_{i\in\text{test}} L\!\left(f(x_i),y_i\right)\]
Held-out empirical risk estimates future error only under a defensible sampling and split design. Repeated runs expose optimization and sampling variability.
Stretch: what the notation leaves out
The estimate is optimistic when test cases leak into training or when the test distribution is easier than the real deployment population.
Interactive explorer
Change assumptions and inspect the decision
Change model complexity and available training data. Watch training fit and expected validation error move differently.
Current signal—
Recommended response—
Increasing complexity usually improves training fit; validation improves only until variance dominates. More representative data can move the useful-complexity region outward.
Evidence workflow
Move from a claim to a decision
Use this order in the chapter, notebook, and project record. Skipping an earlier step weakens every later claim.
- 1. Freeze the task, population, target, baseline, and primary metric.
- 2. Choose a split that blocks group, time, and preprocessing leakage.
- 3. Create an experiment ID with seed, data version, configuration, and code version.
- 4. Repeat the comparison and report a distribution, not only the best run.
- 5. Inspect slices and plausible deployment shifts before selecting a model.
Weekly evidence artifact
Reproducible baseline record with repeated runs
Project connection
Launch PBL1: freeze intended use, target, population, split, baseline, AI-use ledger, and individual responsibility record.
Failure analysis
Deliberately look for the claim's boundary
A failure case is useful when the setup, expected behavior, observation, severity, and response are recorded.
Identity leakage
Failure: Records from the same person appear in training and test data.
Evidence: A grouped split performs much worse than a random split.
Response: Use group-aware validation and state the unit of generalization.
Future information
Failure: A temporal feature contains events occurring after the prediction time.
Evidence: Feature availability audit reveals the timestamp violation.
Response: Rebuild features at the decision cutoff and retest chronologically.
Winner's curse
Failure: Many configurations are tried and only the best validation score is reported.
Evidence: Repeated-run distribution and untouched test performance fail to match the winner.
Response: Limit search, preserve all trials, and use the test set once.
Use and non-use
Keep authority proportional to evidence
Intended use
Use held-out experiments to compare claims under a stated future-use assumption.
Do not use
Do not interpret one random split or one seed as a guarantee of deployment performance.
Human responsibility
A human approves the split rationale, leakage audit, and final claim language.
Small Python demonstrations
Predict, run, and interpret
Each button calls one fixed, allowlisted computation. Use the notebook for longer experiments and saved evidence.
Detect group leakage
train_groups = {"A", "B", "C"}
test_groups = {"C", "D"}
print("overlap:", sorted(train_groups & test_groups))
print("valid grouped split:", not bool(train_groups & test_groups))output
Run this fixed example to compare your prediction with the result.
Summarize repeated runs
import statistics
scores = [0.71, 0.74, 0.69, 0.73, 0.70]
print("mean:", round(statistics.mean(scores), 3))
print("sd:", round(statistics.stdev(scores), 3))output
Run this fixed example to compare your prediction with the result.
Measure a shifted rate
train_rate, future_rate = 0.28, 0.46
print("rate shift:", round(future_rate - train_rate, 2))
print("review required:", abs(future_rate-train_rate) >= 0.10)output
Run this fixed example to compare your prediction with the result.
Check your understanding
Ten questions with standard answers
Answer before opening each panel. A good answer connects the concept to evidence, failure, and a bounded decision.
1. How do we know a model learned something that will transfer?
Standard answer: A model supports a transfer claim only when the split represents future use, leakage is blocked, variability is measured, and failures under relevant shifts are visible.
2. What is the role of generalization gap in this chapter?
Standard answer: The difference between training and held-out behavior can signal overfitting, but a small gap is meaningful only when the held-out sample represents the intended future.
3. Why does split as a causal boundary require evidence rather than intuition?
Standard answer: Random splits assume exchangeable cases. Grouped splits prevent identity leakage; temporal splits prevent learning from the future.
4. How should a practitioner use regularization?
Standard answer: Weight decay, dropout, early stopping, augmentation, and smaller models restrict or disturb fitting. Their value is judged on valid held-out evidence.
5. What does the chapter's main formula clarify—and what does it not prove?
Standard answer: The estimate is optimistic when test cases leak into training or when the test distribution is easier than the real deployment population.
6. What should change in the explorer as its risk or complexity controls increase?
Standard answer: Increasing complexity usually improves training fit; validation improves only until variance dominates. More representative data can move the useful-complexity region outward.
7. How should the system respond to: Identity leakage?
Standard answer: Use group-aware validation and state the unit of generalization.
8. What evidence reveals the failure called Future information?
Standard answer: Feature availability audit reveals the timestamp violation.
9. When should the system not be used or allowed to proceed?
Standard answer: Do not interpret one random split or one seed as a guarantee of deployment performance.
10. How does this week prepare the next stage of the course?
Standard answer: Week 4 asks what architectural assumptions may improve transfer for images and sequences—and when those assumptions are unnecessary.
Terminology
Glossary
- Generalization
- Performance on relevant cases not used to fit or select the model.
- Deployment population
- The people, objects, places, and times where the system will actually operate.
- Validation set
- Held-out data used for model and configuration choices.
- Test set
- Untouched data used for a final estimate after choices are frozen.
- Grouped split
- A split keeping all records from the same entity in one partition.
- Temporal split
- A split training on the past and evaluating on later cases.
- Regularization
- A method that constrains fitting to improve transfer.
- Early stopping
- Stopping training based on held-out behavior before training convergence.
- Distribution shift
- A change between development and deployment data or relationships.
- Experiment registry
- A structured record connecting each run to inputs, code, configuration, and results.
Continue learning
Key references
These primary papers, standards, or official technical documents anchor the chapter. Product names and current legal timelines should be rechecked when used in a real project.