Week 11 · Hands-on foundations

RNN, LSTM, and Sequence Evidence

How do recurrent and gated models use earlier observations without leaking future information?

The historical recurrent unit contains separate notebooks for RNNs, LSTMs, text processing, sentiment, and generation. With PBL presentation weeks restored, this active week keeps the durable comparison: chronological windows, recurrent hidden state, LSTM gates, a naive baseline, and one held-out temporal evaluation. Large NLP pipelines move to the stretch lane.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Create chronological input windows and targets.
Trace a simple recurrent hidden-state update.
Explain how LSTM gates change the memory path.
Compare a naive baseline, RNN, and LSTM using one temporal split.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Use three seeds or a longer synthetic dependency to determine whether an LSTM advantage is stable rather than a one-run accident.

Core ideas

Three concepts to keep

Chronological contract

A sequence example uses earlier values to predict a later value. Splitting by time and fitting preprocessing on the training period prevent future information from entering the model.

Recurrent hidden state

An RNN combines the current input with a previous hidden state using shared parameters. Repeated multiplication can weaken or destabilize long-range influence.

Gated cell state

An LSTM adds gates controlling retained and written information. This changes the path through which information and gradients travel, but it does not remove the need for a baseline and valid split.

Mathematical intuition

One relationship worth keeping

\[\mathbf{c}_t=\mathbf{f}_t\odot\mathbf{c}_{t-1}+\mathbf{i}_t\odot\tilde{\mathbf{c}}_t\]

The LSTM cell retains a gated part of old memory and adds a gated candidate. The simple RNN remains the essential comparison because added memory machinery must improve held-out behavior to be justified.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Old cell state 1Forget gate 0.8Input gate 0.3

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Define what information is available at prediction time.
2. Create windows and split them chronologically.
3. Fit a naive last-value or mean baseline.
4. Train one small RNN and one small LSTM under the same budget.
5. Compare held-out loss and long-gap failures before choosing a model.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Create chronological windows

series = [2, 4, 6, 8, 10]
windows = [(series[i:i+2], series[i+2]) for i in range(3)]
print(windows)

output

Predict the result, then click Run.

Trace a recurrent update

import math
x, previous = 0.5, 0.2
hidden = math.tanh(1.2*x + 0.8*previous)
print(round(hidden, 3))

output

Predict the result, then click Run.

Update an LSTM cell state

old, forget, candidate, input_gate = 1.0, 0.8, 0.5, 0.3
new = forget*old + input_gate*candidate
print(round(new, 2))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Random temporal split

What goes wrong: Nearby future patterns appear in training and make performance optimistic.

Check: Split by time and inspect the last training date and first test date.

Architecture without baseline

What goes wrong: An LSTM is accepted because it is more sophisticated.

Check: Compare against persistence and a simple RNN under identical conditions.

Bidirectional leakage

What goes wrong: A bidirectional model uses future context for a real-time decision.

Check: Match model direction to the actual information boundary.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. How do recurrent and gated models use earlier observations without leaking future information?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. Large NLP pipelines move to the stretch lane.

2. What does chronological contract mean here?

Standard answer: A sequence example uses earlier values to predict a later value. Splitting by time and fitting preprocessing on the training period prevent future information from entering the model.

3. Why is recurrent hidden state useful?

Standard answer: An RNN combines the current input with a previous hidden state using shared parameters. Repeated multiplication can weaken or destabilize long-range influence.

4. How should you interpret gated cell state?

Standard answer: An LSTM adds gates controlling retained and written information. This changes the path through which information and gradients travel, but it does not remove the need for a baseline and valid split.

5. What does the main equation clarify—and what does it not prove?

Standard answer: The LSTM cell retains a gated part of old memory and adds a gated candidate. The simple RNN remains the essential comparison because added memory machinery must improve held-out behavior to be justified. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Define what information is available at prediction time.

8. How can you detect the failure called “Random temporal split”?

Standard answer: Split by time and inspect the last training date and first test date.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 12 replaces recurrent information flow with direct attention across positions and introduces the Transformer block.

Terminology

Glossary

Sequence: Ordered observations whose positions or times matter.
Window: Fixed earlier segment used as one model input.
Hidden state: Recurrent summary passed between steps.
RNN: Network that reuses parameters across an ordered sequence.
LSTM: Gated recurrent architecture with a cell state.
Cell state: Additive recurrent memory path in an LSTM.
Forget gate: Control scaling retained memory.
Input gate: Control scaling a candidate update.
Temporal split: Evaluation split preserving time order.
Persistence baseline: Prediction that repeats the latest observed value.

Go further

Key references

Next: Week 12 replaces recurrent information flow with direct attention across positions and introduces the Transformer block.