Week 12 · Hands-on foundations

Attention and Transformers

How can each position gather information directly from other positions?

The current Transformer materials range from long video playlists and a complete machine-translation build to newer 2025 notebooks that implement self-attention and encoder blocks from scratch. The durable core is smaller: queries, keys, values, scaled dot-product attention, positional information, and a residual feedforward block.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Explain query, key, and value roles with one example.
Calculate and normalize attention scores.
Trace batch × sequence × embedding shapes.
Identify the attention, feedforward, residual, and positional parts of an encoder block.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Add a causal upper-triangular mask and verify that each position receives zero weight from later positions.

Core ideas

Three concepts to keep

Query-key comparison

A query represents what the current position seeks; keys represent what each position offers for matching. Their dot products form relevance scores.

Weighted values

Softmax-normalized scores weight the value vectors. The output is a context-dependent mixture rather than a fixed window or one recurrent state.

Transformer block

Multi-head self-attention is combined with residual paths, normalization, and a position-wise feedforward network. Positional information is added because attention alone does not encode order.

Mathematical intuition

One relationship worth keeping

\[\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\]

Scaling by √d_k prevents dot products from growing too large as key dimension increases. Softmax converts each query's scores into weights over positions.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Similarity to token 1 2Similarity to token 2 1Temperature 1

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Start with three small token vectors and one query.
2. Calculate raw query-key scores by hand.
3. Normalize scores and combine values.
4. Use PyTorch's attention module only after verifying shapes.
5. Interpret attention as a computational weight, not automatically a causal explanation.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Normalize attention scores

import torch
scores=torch.tensor([2.,1.,0.])
weights=torch.softmax(scores,dim=0)
print([round(float(v),3) for v in weights])

output

Predict the result, then click Run.

Combine scalar values

weights=[0.6,0.3,0.1]
values=[10.,2.,-1.]
print(round(sum(w*v for w,v in zip(weights,values)),1))

output

Predict the result, then click Run.

Check score-matrix shape

queries, keys, dimension = 5, 7, 4
print((queries, keys), 'from', (queries,dimension), 'x', (dimension,keys))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Missing scale

What goes wrong: Large dot products make softmax extremely sharp.

Check: Divide by the square root of key dimension.

Mask direction error

What goes wrong: A causal model can attend to future tokens.

Check: Visualize the allowed attention matrix.

Attention-as-explanation

What goes wrong: A high weight is treated as proof of why a model decided.

Check: Use controlled tests and alternative explanation evidence.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. How can each position gather information directly from other positions?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. The durable core is smaller: queries, keys, values, scaled dot-product attention, positional information, and a residual feedforward block.

2. What does query-key comparison mean here?

Standard answer: A query represents what the current position seeks; keys represent what each position offers for matching. Their dot products form relevance scores.

3. Why is weighted values useful?

Standard answer: Softmax-normalized scores weight the value vectors. The output is a context-dependent mixture rather than a fixed window or one recurrent state.

4. How should you interpret transformer block?

Standard answer: Multi-head self-attention is combined with residual paths, normalization, and a position-wise feedforward network. Positional information is added because attention alone does not encode order.

5. What does the main equation clarify—and what does it not prove?

Standard answer: Scaling by √d_k prevents dot products from growing too large as key dimension increases. Softmax converts each query's scores into weights over positions. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Start with three small token vectors and one query.

8. How can you detect the failure called “Missing scale”?

Standard answer: Divide by the square root of key dimension.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 13 turns the learned representation into a transferable backbone and compares frozen features with fine-tuning.

Terminology

Glossary

Attention: Weighted information aggregation based on learned similarity.
Query: Vector used to request relevant information.
Key: Vector compared with a query.
Value: Vector combined using attention weights.
Self-attention: Attention whose queries, keys, and values come from one sequence.
Head: One learned attention projection and aggregation.
Mask: Restriction preventing selected attention links.
Positional encoding: Signal representing sequence position.
Residual connection: Addition of a block input to its transformed output.
Transformer: Architecture built from attention and feedforward blocks.

Go further

Key references

Next: Week 13 turns the learned representation into a transferable backbone and compares frozen features with fine-tuning.