Week 13 Building and Evaluating AI Systems
PBL2 Integration and Adversarial Testing
Integration creates failures that component demos hide. Evidence must be frozen before red-teaming, attacks must be plausible and recorded, and mitigations must be retested without silently changing the evaluation set.
Core question: Can the system survive realistic failure and misuse cases?
By the end
- Freeze a reproducible system version and evidence dossier.
- Design normal, boundary, failure, misuse, and recovery tests.
- Record attack, observed behavior, severity, mitigation, and regression result.
- Prepare a static or recorded fallback and an individual defense.
Course progression
Build one evidence chain
This week uses the course-wide sequence: frame the use, establish a baseline, build or compare, evaluate failure, document boundaries, and decide.
framebaselinebuildevaluatedocumentdecidemonitor
Connection from the course
Week 14 turns the frozen dossier into public demonstration, individual oral accountability, peer evidence review, and a final non-delegation synthesis.
Essential concepts
Understand the parts before combining them
The core lane focuses on transferable judgment. Optional formal or engineering depth belongs in the companion notebook's stretch lane.
Integration contract
Each component defines inputs, outputs, validation, failure behavior, timeout, provenance, and responsibility. Mismatched contracts cause silent failures.
Adversarial test
A deliberate case probes a plausible failure or misuse path. It is evidence when the setup, version, expected boundary, and observation are recorded.
Regression suite
A mitigation is incomplete until earlier successes and failures are rerun. Fixing one case can damage another.
Mathematical intuition
One relationship worth keeping
\[\mathrm{RPN}=\text{likelihood}\times\text{impact}\times\text{difficulty of detection}\]
A risk-priority number helps triage, not certify safety. High-impact cases may require treatment even when their estimated likelihood is low.
Stretch: what the notation leaves out
RPN depends on judgment and can hide uncertainty; report the three ratings, rationale, control, and residual risk rather than only the product.
Interactive explorer
Change assumptions and inspect the decision
Change likelihood, impact, and detection difficulty. Use the score to prioritize—not to declare the system safe.
Current signal—
Recommended response—
The product highlights combinations needing attention, but severe harms, uncertainty, and mandatory controls still require qualitative judgment.
Evidence workflow
Move from a claim to a decision
Use this order in the chapter, notebook, and project record. Skipping an earlier step weakens every later claim.
- 1. Freeze code, model, data, prompt, source, tool, configuration, and evidence versions.
- 2. Create expected behavior for ordinary, boundary, failure, misuse, and recovery cases.
- 3. Run tests without repairing them mid-suite; preserve complete traces.
- 4. Prioritize failures, assign owners, implement bounded mitigations, and rerun regression tests.
- 5. Prepare a live demonstration plus a truthful static or recorded fallback.
Weekly evidence artifact
Frozen evidence dossier, red-team log, mitigation record, and reproducible fallback
Project connection
Freeze PBL2 evidence, run peer red-team and regression tests, assign residual risks, and rehearse individual defenses.
Failure analysis
Deliberately look for the claim's boundary
A failure case is useful when the setup, expected behavior, observation, severity, and response are recorded.
Moving target
Failure: The team changes prompts or data during red-team testing.
Evidence: Results cannot be tied to one system version.
Response: Freeze first; record every mitigation as a new version.
Attack theater
Failure: Only spectacular jailbreak prompts are tested.
Evidence: Ordinary missing data, timeout, stale sources, and repeated-call failures remain unknown.
Response: Use a balanced failure taxonomy tied to the architecture.
Mitigation masking
Failure: A filter blocks the test phrase but not the underlying unsafe action.
Evidence: Paraphrased and indirect variants still succeed.
Response: Reduce authority, validate state, and test mechanism-level controls.
Use and non-use
Keep authority proportional to evidence
Intended use
Use red-teaming to discover and document residual failures in a frozen bounded system.
Do not use
Do not claim safety from a small test suite, hidden repairs, or the absence of one known attack.
Human responsibility
A person owns each risk, approves mitigation, accepts residual exposure, and may stop release.
Small Python demonstrations
Predict, run, and interpret
Each button calls one fixed, allowlisted computation. Use the notebook for longer experiments and saved evidence.
Summarize a test suite
results = [True, True, False, True, False]
print("passed:", sum(results), "of", len(results))
print("pass rate:", round(sum(results)/len(results), 2))output
Run this fixed example to compare your prediction with the result.
Calculate risk priority
likelihood, impact, detection = 3, 5, 4
print("RPN:", likelihood*impact*detection)output
Run this fixed example to compare your prediction with the result.
Fingerprint frozen evidence
import hashlib
manifest = b"model=v3|data=sha256:abc|prompt=v5|tests=24"
print(hashlib.sha256(manifest).hexdigest()[:20])output
Run this fixed example to compare your prediction with the result.
Check your understanding
Ten questions with standard answers
Answer before opening each panel. A good answer connects the concept to evidence, failure, and a bounded decision.
1. Can the system survive realistic failure and misuse cases?
Standard answer: A system earns a defensible claim only after a frozen version survives a balanced, recorded, architecture-specific test suite and mitigations pass regression tests with residual risk visible.
2. What is the role of integration contract in this chapter?
Standard answer: Each component defines inputs, outputs, validation, failure behavior, timeout, provenance, and responsibility. Mismatched contracts cause silent failures.
3. Why does adversarial test require evidence rather than intuition?
Standard answer: A deliberate case probes a plausible failure or misuse path. It is evidence when the setup, version, expected boundary, and observation are recorded.
4. How should a practitioner use regression suite?
Standard answer: A mitigation is incomplete until earlier successes and failures are rerun. Fixing one case can damage another.
5. What does the chapter's main formula clarify—and what does it not prove?
Standard answer: RPN depends on judgment and can hide uncertainty; report the three ratings, rationale, control, and residual risk rather than only the product.
6. What should change in the explorer as its risk or complexity controls increase?
Standard answer: The product highlights combinations needing attention, but severe harms, uncertainty, and mandatory controls still require qualitative judgment.
7. How should the system respond to: Moving target?
Standard answer: Freeze first; record every mitigation as a new version.
8. What evidence reveals the failure called Attack theater?
Standard answer: Ordinary missing data, timeout, stale sources, and repeated-call failures remain unknown.
9. When should the system not be used or allowed to proceed?
Standard answer: Do not claim safety from a small test suite, hidden repairs, or the absence of one known attack.
10. How does this week prepare the next stage of the course?
Standard answer: Week 14 turns the frozen dossier into public demonstration, individual oral accountability, peer evidence review, and a final non-delegation synthesis.
Terminology
Glossary
- Integration contract
- An explicit agreement about a component's inputs, outputs, validation, and failure behavior.
- Evidence freeze
- A fixed version of the system and evidence used for review.
- Adversarial test
- A deliberate probe of a plausible failure or misuse path.
- Threat model
- A structured account of assets, actors, capabilities, entry points, and harms.
- Attack surface
- The components and interfaces through which a system may be misused or compromised.
- Mitigation
- A control reducing likelihood, impact, authority, or detectability of failure.
- Regression test
- A rerun ensuring a change did not break previous behavior.
- Residual risk
- Failure exposure remaining after mitigation.
- Fallback
- A tested alternative when the main system or demonstration fails.
- Evidence dossier
- The complete traceable package supporting system claims and boundaries.
Continue learning
Key references
These primary papers, standards, or official technical documents anchor the chapter. Product names and current legal timelines should be rechecked when used in a real project.