Week 2 · Hands-on foundations

Autograd and Gradient Descent

How does a model know which way to change its parameters?

The current course spends substantial time on PyTorch autograd, computation graphs, optimization, and backpropagation. Those ideas remain essential. The lower-intensity path follows one scalar parameter from prediction to loss, gradient, and update before introducing larger networks.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Describe a forward pass and a loss.
Use `requires_grad`, `backward`, and `.grad` on a tiny example.
Apply one gradient-descent update by hand and in PyTorch.
Distinguish gradient calculation from parameter updating.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Derive and verify the gradient of `(2*w + 1 - target)**2`.

Core ideas

Three concepts to keep

Computation graph

PyTorch records operations connecting tensors that require gradients. Backward traversal applies the chain rule to determine how the final loss changes with each leaf parameter.

Gradient

A gradient is a local sensitivity. Its sign gives a direction of increase; its magnitude gives the local rate of change. It is information, not the update itself.

Learning rate

Gradient descent multiplies the gradient by a chosen step size. A very small rate moves slowly; a very large one can cross the minimum repeatedly or diverge.

Mathematical intuition

One relationship worth keeping

\[w_{t+1}=w_t-\eta\,\frac{\partial L}{\partial w_t}\]

The learning rate η converts local sensitivity into an update. The minus sign moves against the direction in which the loss increases.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Current weight 3Learning rate 0.2Target weight 1

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Make a prediction with current parameters.
2. Compare the prediction with the target using a loss.
3. Clear old gradients before a new update.
4. Call backward to calculate gradients.
5. Update parameters, then repeat and inspect the loss curve.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Differentiate a square

import torch
w = torch.tensor(3.0, requires_grad=True)
loss = (w - 1) ** 2
loss.backward()
print(loss.item(), w.grad.item())

output

Predict the result, then click Run.

Take one update

w, target, rate = 3.0, 1.0, 0.2
grad = 2 * (w - target)
w = w - rate * grad
print(round(w, 2))

output

Predict the result, then click Run.

Compare step sizes

for rate in [0.05, 0.2, 0.8]:
    w = 3.0 - rate * 4.0
    print(rate, round((w - 1) ** 2, 3))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Gradient accumulation

What goes wrong: Repeated backward calls add gradients when they were not reset.

Check: Print the gradient before and after zeroing.

Detached computation

What goes wrong: Conversion to NumPy or `.item()` breaks the graph too early.

Check: Inspect `grad_fn` and keep tensor operations until reporting.

Unstable step

What goes wrong: Loss grows or oscillates.

Check: Compare several learning rates on the same starting state.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. How does a model know which way to change its parameters?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. The lower-intensity path follows one scalar parameter from prediction to loss, gradient, and update before introducing larger networks.

2. What does computation graph mean here?

Standard answer: PyTorch records operations connecting tensors that require gradients. Backward traversal applies the chain rule to determine how the final loss changes with each leaf parameter.

3. Why is gradient useful?

Standard answer: A gradient is a local sensitivity. Its sign gives a direction of increase; its magnitude gives the local rate of change. It is information, not the update itself.

4. How should you interpret learning rate?

Standard answer: Gradient descent multiplies the gradient by a chosen step size. A very small rate moves slowly; a very large one can cross the minimum repeatedly or diverge.

5. What does the main equation clarify—and what does it not prove?

Standard answer: The learning rate η converts local sensitivity into an update. The minus sign moves against the direction in which the loss increases. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Make a prediction with current parameters.

8. How can you detect the failure called “Gradient accumulation”?

Standard answer: Print the gradient before and after zeroing.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 3 embeds the same update inside linear regression, where a weight and bias learn from several observations.

Terminology

Glossary

Forward pass: Computation from input through prediction and loss.
Computation graph: Recorded dependency structure among differentiable operations.
Autograd: PyTorch's automatic differentiation system.
Gradient: Local derivative of an output with respect to a parameter.
Backward pass: Reverse traversal that calculates gradients.
Leaf tensor: A graph input such as a model parameter that can store a gradient.
Loss: A differentiable measure of prediction error.
Learning rate: Multiplier controlling update size.
Optimizer: Procedure that uses gradients to update parameters.
Zero gradient: Resetting stored gradients before the next update.

Go further

Key references

Next: Week 3 embeds the same update inside linear regression, where a weight and bias learn from several observations.