Week 9 Building and Evaluating AI Systems
Vision-Language and Document AI
Multimodal systems can retrieve images with language, caption scenes, and extract structured evidence from documents. Each transformation can also lose layout, uncertainty, provenance, or meaning.
Core question: What is gained and lost when text and images share a representation?
By the end
- Explain contrastive alignment between image and text representations.
- Trace OCR, layout analysis, table extraction, and answer generation separately.
- Audit multimodal evidence at source, extraction, representation, and output layers.
- Design abstention and human review for low-quality or consequential documents.
Course progression
Build one evidence chain
This week uses the course-wide sequence: frame the use, establish a baseline, build or compare, evaluate failure, document boundaries, and decide.
framebaselinebuildevaluatedocumentdecidemonitor
Connection from the course
Week 10 extends component-level multimodal evaluation to speech, audio, and accessibility workflows.
Essential concepts
Understand the parts before combining them
The core lane focuses on transferable judgment. Optional formal or engineering depth belongs in the companion notebook's stretch lane.
Contrastive alignment
Paired image–text examples pull corresponding embeddings together and push mismatched pairs apart. The result supports retrieval, not guaranteed entailment.
Document pipeline
Scanning, OCR, reading order, layout, table extraction, chunking, and answering are distinct components with distinct failure modes.
Multimodal provenance
A useful claim points back to the source file, page, region, extraction method, confidence, and current version.
Mathematical intuition
One relationship worth keeping
\[\mathcal{L}_{\mathrm{contrastive}}=-\log\frac{\exp(s(i,t^+)/\tau)}{\sum_j\exp(s(i,t_j)/\tau)}\]
The loss ranks a matched caption over alternatives. It does not guarantee that every detail in the caption is visible or that the image establishes a consequential claim.
Stretch: what the notation leaves out
A high image–text similarity can reflect broad topic alignment while missing negation, quantities, spatial relations, or fine visual details.
Interactive explorer
Change assumptions and inspect the decision
Change OCR confidence and layout preservation. Decide whether extraction may proceed, needs review, or must abstain.
Current signal—
Recommended response—
Low confidence, lost layout, or high consequence should move the system toward region-level review or abstention even when the text looks fluent.
Evidence workflow
Move from a claim to a decision
Use this order in the chapter, notebook, and project record. Skipping an earlier step weakens every later claim.
- 1. Register the lawful source, page, version, access, and privacy boundary.
- 2. Inspect scan quality, orientation, OCR confidence, and reading order.
- 3. Retain bounding boxes and table structure when layout carries meaning.
- 4. Evaluate extraction before retrieval or answer generation.
- 5. Link every consequential claim to the exact source region and human checkpoint.
Weekly evidence artifact
Multimodal provenance audit across source, extraction, and answer
Project connection
Explore the PBL2 multimodal option and document source, extraction, representation, and human-review boundaries.
Failure analysis
Deliberately look for the claim's boundary
A failure case is useful when the setup, expected behavior, observation, severity, and response are recorded.
Reading-order corruption
Failure: A two-column document is flattened into an incorrect sentence order.
Evidence: Bounding-box visualization contradicts the extracted sequence.
Response: Use layout-aware parsing and region-level inspection.
Table collapse
Failure: Rows and columns are converted to unstructured text.
Evidence: Extracted values lose headers or align with the wrong entity.
Response: Preserve table coordinates and validate totals and keys.
Image-text overreach
Failure: A similar image is treated as evidence about identity or intent.
Evidence: Fine-detail and context checks cannot support the claim.
Response: Limit use to retrieval candidates and require domain review.
Use and non-use
Keep authority proportional to evidence
Intended use
Use multimodal models to organize, extract, and propose evidence with traceable regions.
Do not use
Do not infer identity, intent, eligibility, or other consequential facts from similarity alone.
Human responsibility
A person reviews low-confidence extraction and approves consequential interpretations.
Small Python demonstrations
Predict, run, and interpret
Each button calls one fixed, allowlisted computation. Use the notebook for longer experiments and saved evidence.
Rank image–text pairs
similarities = {"matched": 0.82, "near topic": 0.61, "unrelated": 0.12}
for label, score in sorted(similarities.items(), key=lambda item:item[1], reverse=True):
print(label, score)output
Run this fixed example to compare your prediction with the result.
Gate OCR output
confidences = [0.97, 0.91, 0.58, 0.88]
print("minimum:", min(confidences))
print("human review:", min(confidences) < 0.75)output
Run this fixed example to compare your prediction with the result.
Create a region citation
region = {"file":"policy.pdf", "page":3, "box":[72, 140, 510, 220]}
print(f"{region['file']}#page={region['page']} box={region['box']}")output
Run this fixed example to compare your prediction with the result.
Check your understanding
Ten questions with standard answers
Answer before opening each panel. A good answer connects the concept to evidence, failure, and a bounded decision.
1. What is gained and lost when text and images share a representation?
Standard answer: Shared representations enable cross-modal retrieval and processing, but reliable document AI must preserve layout, confidence, exact source regions, and component-level evaluation.
2. What is the role of contrastive alignment in this chapter?
Standard answer: Paired image–text examples pull corresponding embeddings together and push mismatched pairs apart. The result supports retrieval, not guaranteed entailment.
3. Why does document pipeline require evidence rather than intuition?
Standard answer: Scanning, OCR, reading order, layout, table extraction, chunking, and answering are distinct components with distinct failure modes.
4. How should a practitioner use multimodal provenance?
Standard answer: A useful claim points back to the source file, page, region, extraction method, confidence, and current version.
5. What does the chapter's main formula clarify—and what does it not prove?
Standard answer: A high image–text similarity can reflect broad topic alignment while missing negation, quantities, spatial relations, or fine visual details.
6. What should change in the explorer as its risk or complexity controls increase?
Standard answer: Low confidence, lost layout, or high consequence should move the system toward region-level review or abstention even when the text looks fluent.
7. How should the system respond to: Reading-order corruption?
Standard answer: Use layout-aware parsing and region-level inspection.
8. What evidence reveals the failure called Table collapse?
Standard answer: Extracted values lose headers or align with the wrong entity.
9. When should the system not be used or allowed to proceed?
Standard answer: Do not infer identity, intent, eligibility, or other consequential facts from similarity alone.
10. How does this week prepare the next stage of the course?
Standard answer: Week 10 extends component-level multimodal evaluation to speech, audio, and accessibility workflows.
Terminology
Glossary
- Multimodal
- Using or relating more than one data modality, such as text and images.
- Contrastive learning
- Learning representations by ranking matched pairs above mismatches.
- Vision-language model
- A model connecting visual and language representations or generation.
- OCR
- Optical character recognition that converts image regions into text candidates.
- Reading order
- The intended sequence of document regions.
- Layout analysis
- Detection of regions such as headings, columns, figures, and tables.
- Bounding box
- Coordinates locating a region in an image or page.
- Region citation
- A reference to the exact page area supporting a claim.
- Extraction confidence
- A component estimate of recognition reliability, not truth.
- Multimodal provenance
- Traceable links across source media, transformations, and output claims.
Continue learning
Key references
These primary papers, standards, or official technical documents anchor the chapter. Product names and current legal timelines should be rechecked when used in a real project.