Week 6 · Hands-on foundations

Classification Workflow and Evaluation

Which evidence shows whether a classifier is useful for its intended decision?

The later classification notebook combines deeper networks, dropout, learning-rate schedules, early stopping, hyperparameter search, exercises, and project groups. The new core omits private group information and optional Optuna machinery. It concentrates on a reproducible pipeline and the error trade-offs students must interpret.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Build a stratified train/validation/test split.
Read a confusion matrix by actual and predicted class.
Calculate precision and recall from counts.
Choose a threshold from error consequences rather than habit.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Add a cost table and choose the threshold minimizing total validation cost.

Core ideas

Three concepts to keep

Confusion matrix

A confusion matrix keeps correct and incorrect outcomes separated by class. It makes the direction of a mistake visible, which a single accuracy value cannot.

Precision and recall

Precision asks how often positive predictions are correct. Recall asks how many actual positives were found. They answer different operational questions.

Threshold

A binary classifier can convert a score or probability to an action using a threshold. Changing the threshold changes the mix of false positives and false negatives without retraining.

Mathematical intuition

One relationship worth keeping

\[\mathrm{precision}=\frac{TP}{TP+FP},\qquad \mathrm{recall}=\frac{TP}{TP+FN}\]

Precision conditions on predicted positives; recall conditions on actual positives. The denominators encode the question each metric answers.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Threshold 0.5False-negative cost 8False-positive cost 3

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. State which class is treated as positive and why.
2. Split with class balance and identity/time boundaries in mind.
3. Train a simple baseline before a deeper network.
4. Inspect confusion counts, precision, recall, and class support.
5. Choose a threshold and record the error trade-off it creates.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Read confusion counts

actual = [1, 1, 1, 0, 0, 0]
pred =   [1, 0, 1, 1, 0, 0]
tp = sum(a==1 and p==1 for a,p in zip(actual,pred))
fp = sum(a==0 and p==1 for a,p in zip(actual,pred))
fn = sum(a==1 and p==0 for a,p in zip(actual,pred))
print(tp, fp, fn)

output

Predict the result, then click Run.

Calculate precision and recall

tp, fp, fn = 2, 1, 1
print(round(tp/(tp+fp), 3), round(tp/(tp+fn), 3))

output

Predict the result, then click Run.

Apply a threshold

scores = [0.2, 0.49, 0.51, 0.8]
threshold = 0.5
print([int(s >= threshold) for s in scores])

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Metric without positive class

What goes wrong: Precision is reported without stating which outcome is positive.

Check: Name the decision and class explicitly.

Unstratified small split

What goes wrong: One subset contains very few examples of a class.

Check: Inspect counts after splitting.

Threshold tuned on test

What goes wrong: The final estimate becomes optimistic.

Check: Select the threshold on validation data only.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. Which evidence shows whether a classifier is useful for its intended decision?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. It concentrates on a reproducible pipeline and the error trade-offs students must interpret.

2. What does confusion matrix mean here?

Standard answer: A confusion matrix keeps correct and incorrect outcomes separated by class. It makes the direction of a mistake visible, which a single accuracy value cannot.

3. Why is precision and recall useful?

Standard answer: Precision asks how often positive predictions are correct. Recall asks how many actual positives were found. They answer different operational questions.

4. How should you interpret threshold?

Standard answer: A binary classifier can convert a score or probability to an action using a threshold. Changing the threshold changes the mix of false positives and false negatives without retraining.

5. What does the main equation clarify—and what does it not prove?

Standard answer: Precision conditions on predicted positives; recall conditions on actual positives. The denominators encode the question each metric answers. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: State which class is treated as positive and why.

8. How can you detect the failure called “Metric without positive class”?

Standard answer: Name the decision and class explicitly.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 7 replaces the linear classifier with hidden layers while keeping exactly the same data split and evaluation contract.

Terminology

Glossary

Confusion matrix: Table of actual classes against predicted classes.
True positive: Positive case correctly predicted positive.
False positive: Negative case incorrectly predicted positive.
False negative: Positive case incorrectly predicted negative.
Precision: Fraction of positive predictions that are correct.
Recall: Fraction of actual positives detected.
Specificity: Fraction of actual negatives correctly rejected.
Threshold: Cutoff converting a score to a class or action.
Stratification: Preserving class proportions when splitting.
Support: Number of observed examples for a class.

Go further

Key references

Next: Week 7 replaces the linear classifier with hidden layers while keeping exactly the same data split and evaluation contract.