Week 5 Building and Evaluating AI Systems

Evaluation Before Application

Evaluation is the disciplined connection between model behavior and a real use. It begins by defining the population, decision, and consequences; it ends only after uncertainty, failures, and operating conditions are visible.

Core question: What evidence would justify trusting this system for its intended use?

1 · DefineName intended use, non-use, population, action, and error consequences.

2 · MeasureUse leakage-safe held-out data, multiple metrics, uncertainty, calibration, and slices.

3 · DecideSelect a threshold, referral rule, and monitoring plan—or do not release.

By the end

Design a split that represents the intended future use and prevents group, time, or preprocessing leakage.
Derive precision, recall, specificity, and error rates from a confusion matrix.
Separate discrimination, probability calibration, threshold policy, and uncertainty.
Audit slices, abstention, and drift before making a bounded release claim.

Define

Evaluation begins before the metric

A score has no stable meaning until it is connected to an intended population and action. The same model may be acceptable for queue prioritization and unacceptable for automatic denial.

Five-part evaluation contract

Population: who or what may be represented, and who is outside scope?
Target: what observable outcome is predicted, over what time horizon?
Action: what happens after a score, and who decides?
Consequences: who bears false-positive, false-negative, delay, and abstention costs?
Boundary: where must the system not be used?

Fictional case · support-request triage

Use: prioritize which non-emergency requests receive early human review.

Non-use: never deny service automatically or handle safety-critical emergencies.

Asymmetry: for exploration, a missed urgent request receives five times the teaching cost of an unnecessary early review.

Human boundary: uncertain cases are referred; final action remains human.

Target validity precedes predictive validity

A conveniently recorded label can be a poor proxy for the construct of interest. Before modeling, ask how the label was produced, which decisions shaped it, who is missing, and whether predicting it would reproduce an earlier institutional process.

Evidence design

A test set is a claim about the future

The split must reproduce the independence and change expected after release. Reserve the test set for the final estimate; use training data for fitting and validation data for model and threshold choices.

Random split

Reasonable only when observations are exchangeable and no person, group, document family, or future-derived feature crosses partitions.

Group split

Keep all records from one person, institution, author, or other correlated unit in one partition.

Time split

Train on earlier data and evaluate on later data when deployment predicts forward through changing conditions.

Common leakage paths

Scaling, imputation, feature selection, or text vocabulary fitted before the split.
Near-duplicate documents or repeated people placed in both training and test data.
Features recorded after the target event or generated using the target.
Repeated inspection of the test set while choosing models or thresholds.

Leakage invalidates the estimate

Leakage is not merely optimistic performance. It changes what experiment was conducted: the measured system has access to information the deployed system will not possess.

Measure

Start from counts, then compress carefully

For a chosen threshold, the confusion matrix records true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Each metric answers a different conditional question.

Recall / sensitivity

\[\operatorname{Recall}=\frac{TP}{TP+FN}\]

Among actual positives, how many did the policy identify?

Precision

\[\operatorname{Precision}=\frac{TP}{TP+FP}\]

Among positive actions, how many were warranted?

Specificity

\[\operatorname{Specificity}=\frac{TN}{TN+FP}\]

Among actual negatives, how many were correctly left negative?

\[\operatorname{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN},\qquad F_1=2\frac{\operatorname{Precision}\operatorname{Recall}}{\operatorname{Precision}+\operatorname{Recall}}\]

No single metric is the evidence

Accuracy can look strong under class imbalance. (F_1) ignores true negatives and silently weights precision and recall equally. ROC AUC summarizes ranking across many thresholds, not performance at the chosen operating point. Always report the counts, data support, and decision consequences alongside summaries.

Recover metrics from held-out predictions

from sklearn.metrics import confusion_matrix, precision_score, recall_score

y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
y_pred = [1, 1, 1, 0, 1, 1, 0, 0, 0, 0]

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print("TN FP FN TP:", tn, fp, fn, tp)
print("precision:", round(precision_score(y_true, y_pred), 3))
print("recall:", round(recall_score(y_true, y_pred), 3))

output

Click Run to see the results.

Decide

A threshold is a decision policy

A model produces scores; a threshold maps those scores to actions. Moving it changes false positives, false negatives, workload, and coverage without retraining the model.

\[\widehat{y}=\mathbb{1}[p\ge t],\qquad C(t)=5FN(t)+FP(t)+A(t)\]

The cost (C(t)) is a transparent teaching assumption, not an official social valuation. Real weights require stakeholder participation and study of downstream harm.

	Predict positive	Predict negative

Slice check under the same policy

Slice	Recall	False-positive rate	Abstain

Threshold and referral explorer

Action threshold 0.50Abstention half-band 0.00

recall—

precision—

coverage—

teaching cost—

Change the policy to inspect consequences.

Compare operating points on fixed probabilities

from sklearn.metrics import precision_score, recall_score

y_true = [1, 1, 1, 0, 0, 0, 0, 1]
probability = [0.91, 0.72, 0.55, 0.61, 0.43, 0.32, 0.08, 0.47]

for threshold in [0.30, 0.50, 0.70]:
    prediction = [int(p >= threshold) for p in probability]
    print(threshold,
          "precision", round(precision_score(y_true, prediction), 2),
          "recall", round(recall_score(y_true, prediction), 2))

output

Click Run to see the results.

Probability quality

Discrimination and calibration are different

Discrimination asks whether positives tend to receive higher scores than negatives. Calibration asks whether stated probabilities agree with observed frequencies over a defined population and time window.

\[\operatorname{Brier}=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2\]

A reliability diagram groups comparable predictions and plots mean predicted probability against observed event frequency. Perfect calibration lies near the diagonal, but small bins can be noisy.

Calibration must be learned on data separate from model fitting and checked again after population or process change. A calibrated score is still not causal and does not determine an acceptable threshold.

Further exploration · proper scoring rules

Brier and logarithmic loss are proper scoring rules: honest probability estimates minimize expected score. Their overall values still combine aspects of reliability, resolution, and irreducible uncertainty, so inspect the reliability curve and score distribution as well.

Confidence transformation

Probability behavior

Brier loss: —

Bin	Mean predicted	Observed rate	n

Choose a transformation.

Inspect Brier loss and reliability bins

from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

y_true = [0, 0, 0, 1, 0, 1, 1, 1, 1, 1]
probability = [0.08, 0.18, 0.31, 0.37, 0.44, 0.57, 0.66, 0.73, 0.84, 0.93]

observed, predicted = calibration_curve(y_true, probability, n_bins=4)
print("Brier:", round(brier_score_loss(y_true, probability), 3))
for mean_p, event_rate in zip(predicted, observed):
    print(round(mean_p, 2), "->", round(event_rate, 2))

output

Click Run to see the results.

Uncertainty

A point estimate is not a stability claim

Performance varies because of finite samples, random model fitting, and genuine change in the population. Repeat relevant experiments and show the distribution, not only the best run.

Sampling uncertainty

Bootstrap the independent evaluation records or use an appropriate interval. Preserve grouping or time structure when resampling.

Training variation

Repeat random seeds when initialization, mini-batches, or optimization are stochastic. Report mean, spread, and failures.

Comparison uncertainty

Compare models on the same cases. Paired differences are usually more informative than independent headline scores.

An interval quantifies variation under stated assumptions; it does not correct leakage, target bias, unmeasured population shift, or a poor decision contract.

Failure visibility

Averages can hide concentrated failure

Evaluate predeclared groups connected to access, language, setting, device, time, or another plausible failure mechanism. Also inspect intersectional groups when sample support and privacy allow.

The acceptable average

Claim: “Overall recall is high enough for release.”

Counter-evidence: one language-context slice has substantially lower recall and already faces higher service barriers.

Next step: verify sample support, investigate data and process causes, report uncertainty, and revise or restrict release. Do not hide the result by dropping the slice.

Slice analysis is diagnostic, not a guarantee of fairness. Tiny groups create unstable estimates and privacy risks; combine quantitative evidence with process knowledge and affected-stakeholder review.

Human boundary

“I do not know” can be a designed output

Abstention trades automatic coverage for an opportunity to reduce high-consequence errors—only if a qualified and adequately resourced review path exists.

\[\operatorname{Coverage}=\frac{\text{automatically decided cases}}{\text{eligible cases}}\]

Confidence rule

Refer scores near the action boundary or cases beyond the validated population.

Capacity rule

Measure reviewer workload and waiting time; otherwise abstention merely converts errors into delay.

Outcome rule

Evaluate the complete human–AI workflow, including referral outcomes and who is disproportionately referred.

After release

Evaluation becomes monitoring

Deployment changes inputs, behavior, and incentives. Define what will be measured, how outcomes arrive, and which signal triggers investigation, rollback, or retirement.

Input drift

Track schema violations, missingness, score distributions, and meaningful population or language changes.

Performance drift

When delayed labels arrive, re-estimate errors, calibration, slices, and referral outcomes using the current operating point.

Process drift

Watch how people adapt to the system, whether overrides change, and whether the original action or target remains valid.

Precommit to response

A dashboard without an owner or action threshold is observation, not governance. Record version, data window, threshold, known limits, alert rule, responsible person, and a reversible fallback.

Self-check

Try before revealing the answer

1. Why must intended use be written before selecting metrics?

Metrics summarize particular errors. Intended use identifies the population, action, and consequences that determine which errors matter and where the system must not operate.

2. Why can a random record split be invalid for repeated users?

Records from the same user can appear on both sides, letting the model exploit person-specific information. A group split estimates transfer to genuinely unseen users more honestly.

3. If (TP=30) and (FN=10), what is recall?

Recall is (30/(30+10)=0.75). It says nothing by itself about false positives or probability quality.

4. Lowering a threshold improves recall. Is the new threshold better?

Not from recall alone. Inspect false positives, error consequences, slices, workload, calibration, and the intended action.

5. Can ROC AUC identify the threshold to deploy?

No. ROC AUC summarizes ranking over many thresholds. Deployment still requires an operating point chosen from consequences, constraints, calibration, and the relevant population.

6. A model has Brier loss 0.12 rather than 0.15. Is it necessarily better calibrated?

No. Brier loss combines calibration, resolution, and uncertainty. Inspect reliability on independent data as well.

7. Why report results across random seeds?

Optimization and initialization can change fitted behavior. Seed variation reveals whether a conclusion is stable rather than the product of one favorable run.

8. Overall performance is acceptable but one important slice is poor. What follows?

The aggregate release claim is insufficient. Check support and uncertainty, investigate causes, and improve or restrict the use instead of hiding the slice.

9. Coverage falls after adding abstention while decided-case accuracy rises. What must be reported?

Report both coverage and decided-case performance, plus referral workload, delay, outcomes, and which groups are more often referred.

10. Why is a one-time test insufficient after deployment?

Inputs, processes, and user behavior change. Monitoring must detect drift and connect it to a named response such as investigation, rollback, or retirement.

Continue in Python

Evaluation harness and model-behavior audit

The notebook turns these concepts into a reproducible evidence table with calibration, slice checks, repeated trials, abstention analysis, and a failure taxonomy.

Essential path

Compare a transparent baseline and a flexible model on fictional generated data. Freeze an evidence table before making a release claim.

Evidence artifact: metrics, counts, slices, repeated-run variation, failure categories, and one defended threshold/referral choice.

Further exploration

Bootstrap a paired metric difference and calibrate a flexible model without reusing its fitting data.

Glossary

Key terminology

Intended use: The declared population, purpose, action, and operating conditions for a system.
Target: The observed outcome a model is trained or evaluated to predict.
Data leakage: Information unavailable at deployment entering model fitting or evaluation.
Confusion matrix: Counts of true/false positives and true/false negatives at a threshold.
Discrimination: The ability to rank positive cases above negative cases.
Calibration: Agreement between stated probabilities and observed frequencies.
Operating point: The threshold and associated behavior selected for use.
Slice: A meaningful subgroup or context examined separately for failures.
Abstention: A designed decision not to predict automatically, usually paired with referral.
Coverage: The proportion of eligible cases receiving an automatic decision.
Drift: Change in inputs, relationships, outcomes, or processes over time.
Evaluation harness: Reusable code and data contracts that produce comparable evidence across systems.

References

Key references

scikit-learn User Guide: Metrics and scoring.
scikit-learn User Guide: Tuning the decision threshold.
scikit-learn User Guide: Probability calibration.
NIST AI Risk Management Framework.
The displayed records and cost rule are original fictional teaching data, not an official risk model or institutional policy.