Week 10 Building and Evaluating AI Systems

Speech, Audio, Accessibility, and Multimodal Evaluation

Speech systems transform a time-varying signal into text, meaning, or synthesized audio. A useful evaluation separates signal quality, transcription errors, semantic impact, subgroup behavior, and accessibility.

Core question: How should we evaluate transformations among speech, text, and meaning?

SignalSampling and acoustic conditions define the observable input.

TransformASR, language processing, and TTS introduce different errors.

Make accessibleProvide synchronized captions, transcripts, alternatives, and correction routes.

By the end

Connect sampling, features, transcription, and synthesis as separate stages.
Compute word error rate and inspect which errors change meaning.
Evaluate noise, language, accent, pace, and assistive-use slices.
Design transcripts, captions, alternatives, and human correction paths.

Course progression

Build one evidence chain

This week uses the course-wide sequence: frame the use, establish a baseline, build or compare, evaluate failure, document boundaries, and decide.

framebaselinebuildevaluatedocumentdecidemonitor

Connection from the course

Week 11 asks when a probabilistic model may move beyond transforming information and call tools that affect a workflow.

Essential concepts

Understand the parts before combining them

The core lane focuses on transferable judgment. Optional formal or engineering depth belongs in the companion notebook's stretch lane.

Sampling and representation

A waveform becomes discrete samples and time-frequency features. More samples increase detail and cost but do not repair a poor microphone or missing signal.

Word error rate

WER counts substitutions, deletions, and insertions relative to reference words. It should be paired with semantic and slice analysis.

Accessibility workflow

Captions, transcripts, audio descriptions, keyboard access, and human correction are system requirements rather than optional polish.

Mathematical intuition

One relationship worth keeping

\[\mathrm{WER}=\frac{S+D+I}{N}\]

The same WER can hide very different consequences. Errors in names, numbers, negation, or instructions can matter far more than filler-word differences.

Stretch: what the notation leaves out

WER treats edits uniformly, so it must be supplemented with semantic severity, language/accent slices, and task-level success.

Interactive explorer

Change assumptions and inspect the decision

Set substitutions, deletions, insertions, and reference length. Interpret WER alongside the likely meaning cost.

Substitutions 2Deletions 1Insertions 1Reference words 20

Current signal—

Recommended response—

WER rises with edit count and falls with reference length, but a low WER can still be unacceptable when one error changes a name, number, negation, or instruction.

Evidence workflow

Move from a claim to a decision

Use this order in the chapter, notebook, and project record. Skipping an earlier step weakens every later claim.

1. Obtain consent and register audio source, purpose, retention, and access.
2. Record sampling, channel, language, noise, and assistive-use conditions.
3. Evaluate ASR edits and semantic severity by relevant slices.
4. Evaluate TTS intelligibility, naturalness, pronunciation, and harmful misreading separately.
5. Provide corrected transcript/caption, alternative modality, and human escalation.

Weekly evidence artifact

Accessibility workflow with modality-specific evaluation, human checkpoint, and fallback

Project connection

Explore the PBL2 speech/accessibility option with consent, slice evaluation, correction, and fallback evidence.

Failure analysis

Deliberately look for the claim's boundary

A failure case is useful when the setup, expected behavior, observation, severity, and response are recorded.

Average-only WER

Failure: A strong aggregate score hides poor performance for one language or acoustic condition.

Evidence: Slice WER and semantic error counts reveal the gap.

Response: Report slices and redesign or constrain the supported conditions.

Meaning-changing error

Failure: A transcript drops a negation or changes a number.

Evidence: Task-level comparison shows the decision meaning changed.

Response: Flag critical terms and require human verification.

Accessibility afterthought

Failure: Generated audio is published without synchronized captions or a transcript.

Evidence: Users cannot access or correct the same information.

Response: Build equivalent modalities and correction into the release contract.

Use and non-use

Keep authority proportional to evidence

Intended use

Use speech systems as assistive transformations when users can inspect and correct outputs.

Do not use

Do not use voice or speech patterns to infer protected, medical, emotional, or identity traits without a lawful validated purpose.

Human responsibility

A person reviews meaning-critical content and owns consent, accessibility, retention, and escalation.

Small Python demonstrations

Predict, run, and interpret

Each button calls one fixed, allowlisted computation. Use the notebook for longer experiments and saved evidence.

Compute word error rate

substitutions, deletions, insertions, reference_words = 2, 1, 1, 20
wer = (substitutions + deletions + insertions) / reference_words
print("WER:", round(wer, 2))

output

Run this fixed example to compare your prediction with the result.

Count audio samples

sample_rate, seconds = 16000, 2.5
print("samples:", int(sample_rate * seconds))

output

Run this fixed example to compare your prediction with the result.

Apply a critical-term gate

reference = {"not", "17:30"}
transcript = {"17:30"}
missing = sorted(reference - transcript)
print("missing:", missing)
print("human review:", bool(missing))

output

Run this fixed example to compare your prediction with the result.

Check your understanding

Ten questions with standard answers

Answer before opening each panel. A good answer connects the concept to evidence, failure, and a bounded decision.

1. How should we evaluate transformations among speech, text, and meaning?

Standard answer: Evaluate each transformation separately, then test semantic and accessibility outcomes by relevant conditions; aggregate WER alone is not sufficient.

2. What is the role of sampling and representation in this chapter?

Standard answer: A waveform becomes discrete samples and time-frequency features. More samples increase detail and cost but do not repair a poor microphone or missing signal.

3. Why does word error rate require evidence rather than intuition?

Standard answer: WER counts substitutions, deletions, and insertions relative to reference words. It should be paired with semantic and slice analysis.

4. How should a practitioner use accessibility workflow?

Standard answer: Captions, transcripts, audio descriptions, keyboard access, and human correction are system requirements rather than optional polish.

5. What does the chapter's main formula clarify—and what does it not prove?

Standard answer: WER treats edits uniformly, so it must be supplemented with semantic severity, language/accent slices, and task-level success.

6. What should change in the explorer as its risk or complexity controls increase?

Standard answer: WER rises with edit count and falls with reference length, but a low WER can still be unacceptable when one error changes a name, number, negation, or instruction.

7. How should the system respond to: Average-only WER?

Standard answer: Report slices and redesign or constrain the supported conditions.

8. What evidence reveals the failure called Meaning-changing error?

Standard answer: Task-level comparison shows the decision meaning changed.

9. When should the system not be used or allowed to proceed?

Standard answer: Do not use voice or speech patterns to infer protected, medical, emotional, or identity traits without a lawful validated purpose.

10. How does this week prepare the next stage of the course?

Standard answer: Week 11 asks when a probabilistic model may move beyond transforming information and call tools that affect a workflow.

Terminology

Glossary

Waveform: Signal amplitude represented over time.
Sample rate: Number of signal samples recorded per second.
Spectrogram: A time–frequency representation of a signal.
ASR: Automatic speech recognition that proposes text from audio.
TTS: Text-to-speech synthesis that generates audio from text.
WER: Word error rate: substitutions plus deletions plus insertions divided by reference words.
Semantic error: An output difference that changes relevant meaning.
Caption: Synchronized text for speech and meaningful non-speech audio.
Transcript: A text alternative to audio or audiovisual content.
Assistive technology: Technology supporting access for people with disabilities.

Continue learning

Key references

These primary papers, standards, or official technical documents anchor the chapter. Product names and current legal timelines should be rechecked when used in a real project.