Week 2 Building and Evaluating AI Systems
Neural Foundations
A neural network learns by making a prediction, measuring a loss, tracing how each parameter affected that loss, and applying a bounded update. This chapter makes that loop visible before the notebook automates it.
Core question: What actually changes during learning—and what evidence tells us whether the change is useful?
By the end
- Read tensor shapes as a data contract rather than as incidental syntax.
- Explain the roles of a forward pass, loss, backward pass, and optimizer update.
- Use the sign and magnitude of a gradient to predict an update direction.
- Distinguish evidence of optimization from evidence of generalization.
Represent
Tensors are shaped data contracts
A tensor is an array with a shape and data type. In a learning system, those two facts tell us what each axis means and which operations are valid.
Interpret the axes
A feature table with 400 rows and 2 columns has shape (400, 2): 400 observations, two measured features. A weight matrix with shape (2, 8) maps those two features into eight hidden activations.
\[X_{400\times2}W^{(1)}_{2\times8}+b^{(1)}_{8}\longrightarrow H_{400\times8}\]
Shapes expose many errors before training begins. If the inner dimensions do not agree, the intended transformation is not defined. If they agree for the wrong reason, the code may run while the data meaning is still wrong.
Stretch lane · broadcasting and batch axes
Broadcasting lets a bias vector with shape (8,) act across all 400 rows of H. This is convenient, but it can also hide a missing or misplaced batch axis. Before accepting a result, state the meaning of every axis.
Shape reasoning check
| Object | Shape | Meaning |
|---|---|---|
X | 400 × 2 | observations × features |
W₁ | 2 × 8 | input features × hidden units |
b₁ | 8 | one bias per hidden unit |
W₂ | 8 × 1 | hidden units × output score |
Check: the complete batch output has shape (400, 1). It contains one score per observation—not one score for the whole dataset.
Tensor shapes through a small network
import torch
X = torch.randn(400, 2)
layer1 = torch.nn.Linear(2, 8)
layer2 = torch.nn.Linear(8, 1)
hidden = torch.relu(layer1(X))
scores = layer2(hidden)
print(X.shape, hidden.shape, scores.shape)output
Click Run to see the results.
Trace
A forward pass makes a claim
For one binary prediction, a compact model combines an input with a weight and bias, converts the score to a probability, and compares that probability with the observed target.
\[z=wx+b,\qquad p=\sigma(z)=\frac{1}{1+e^{-z}}\]
\[\mathcal{L}_{\mathrm{BCE}}(y,p)=-\left[y\log p+(1-y)\log(1-p)\right]\]
The score z can be any real number. The sigmoid maps it to a value between zero and one. Binary cross-entropy assigns a small loss when the predicted probability agrees with the target and a large loss when a confident prediction is wrong.
Loss is an objective, not the whole evaluation
Training loss tells the optimizer what to reduce on the training data. It does not establish transfer, calibration, fairness across slices, or suitability for a real decision.
Computational graph explorer
xwz=0ploss∂L/∂wupdate
probability p0
loss L0
gradient ∂L/∂w0
Move a control to trace the claim.
Backward
A gradient answers a local counterfactual
The gradient says how the loss would change under a very small parameter change, holding the current example and computation fixed.
Which direction?
If ∂L/∂w is positive, a small increase in w would increase loss, so gradient descent moves w downward.
How sensitive?
A larger absolute gradient means the present loss is more sensitive to this parameter locally. It is not a universal measure of importance.
For which data?
A mini-batch gradient aggregates current examples. Change the batch, objective, or model state and the gradient may change.
\[w_{t+1}=w_t-\eta\frac{\partial\mathcal{L}}{\partial w_t}\]
Stretch lane · chain rule behind autograd
The dependency path is w → z → p → L. The chain rule multiplies local rates of change along that path:
\[\frac{\partial\mathcal{L}}{\partial w}=\frac{\partial\mathcal{L}}{\partial p}\frac{\partial p}{\partial z}\frac{\partial z}{\partial w}=(p-y)x\]
Automatic differentiation records operations during the forward pass and applies these local derivative rules backward. It computes gradients; it does not decide whether the objective is appropriate.
Inspect one automatic-differentiation step
import torch
x = torch.tensor([1.2])
y = torch.tensor([1.0])
w = torch.tensor([-0.8], requires_grad=True)
b = torch.tensor([0.2], requires_grad=True)
probability = torch.sigmoid(w * x + b)
loss = torch.nn.functional.binary_cross_entropy(probability, y)
loss.backward()
print("probability:", probability.item())
print("dL/dw:", w.grad.item())output
Click Run to see the results.
Compose
From one unit to a multilayer perceptron
A multilayer perceptron (MLP) alternates affine transformations and nonlinear activation functions. The hidden units learn intermediate representations that can bend a linear decision boundary.
\[h=\phi\!\left(XW^{(1)}+b^{(1)}\right),\qquad \hat{y}=g\!\left(hW^{(2)}+b^{(2)}\right)\]
Without the nonlinear function \(\phi\), stacking linear layers still produces one linear transformation. ReLU, \(\operatorname{ReLU}(z)=\max(0,z)\), gives the network piecewise-linear flexibility while remaining simple to differentiate.
Width controls how many hidden features can be represented. Depth controls how many transformations are composed. More capacity can improve fit, but it also increases optimization difficulty, variance across runs, and the opportunity to memorize noise.
Architecture is only one hypothesis
Begin with a transparent baseline. Change one factor at a time, repeat random seeds, and judge the network on held-out data. A larger network is not automatically a better system.
Optional mathematics · parameter count
For an input dimension \(d\), hidden width \(h\), and one output, the two affine layers contain \(dh+h+h+1=h(d+2)+1\) trainable parameters. Biases count because they are learned too.
Test
An optimizer turns gradients into a trajectory
A learning rate sets the update scale. Too small can waste time; too large can overshoot. The useful value depends on the objective, parameterization, data scale, and optimizer.
Training is a sequence, not one backward pass:
- 1. Clear gradients left from the previous step.
- 2. Predict with the current parameters.
- 3. Measure the loss for the current batch.
- 4. Differentiate the loss with respect to parameters.
- 5. Update parameters using the optimizer rule.
- 6. Record enough evidence to diagnose the trajectory.
Learning-rate explorer
Move the learning rate to compare trajectories.
Compare three learning rates on the same objective
import torch
for learning_rate in [0.02, 0.30, 4.00]:
w = torch.tensor([-2.2], requires_grad=True)
for step in range(24):
loss = -torch.log(torch.sigmoid(1.4 * w))
loss.backward()
with torch.no_grad():
w -= learning_rate * w.grad
w.grad.zero_()
print(learning_rate, round(loss.item(), 4))output
Click Run to see the results.
Failure case
A falling training loss can support the wrong story
The memorizing network
Claim: “Loss fell almost to zero, so the network learned the task.”
What the trace supports: the optimizer found parameters that fit the observed training examples under the chosen objective.
What is missing: performance on held-out data, repeated-run stability, leakage checks, class/slice behavior, calibration, and evidence that the metric matches the intended use.
Decision: keep the training trace as optimization evidence, but do not call the system useful until a valid evaluation supports transfer.
Reasoning check
Try before revealing
1. A parameter has gradient −0.6. With learning rate 0.1, which way does gradient descent move it?
It increases the parameter by 0.06 because \(w\leftarrow w-0.1(-0.6)\). This is a local update, not evidence that the next loss must fall for every example.
2. Training loss falls smoothly, but validation loss rises. What has been established?
Optimization is working on the training objective, while transfer is worsening. The pattern is consistent with overfitting or train/validation mismatch and requires investigation.
3. Why is a gradient not automatically an explanation of model behavior?
It is local to the current input, parameters, and objective. It measures sensitivity, not causal importance, fairness, or suitability for the intended decision.
4. What does a tensor shape of (64, 20) usually mean in a mini-batch?
It commonly means 64 observations and 20 features per observation. The interpretation must still be verified from the data contract rather than guessed from the numbers.
5. Why does an MLP need a nonlinear activation between linear layers?
Without nonlinearity, the composition of linear layers collapses to one linear transformation. A nonlinear activation lets the network represent curved or piecewise-linear relationships.
6. What is the difference between a score and a probability in binary classification?
The score or logit may be any real number. The sigmoid maps it into the interval from zero to one, which can be interpreted as a model probability only after suitable evaluation and calibration.
7. Why are gradients cleared between optimization steps?
PyTorch accumulates gradients by default. Clearing them prevents the next update from unintentionally adding the previous mini-batch gradient.
8. Can a very large gradient prove that a feature is globally important?
No. It shows local sensitivity for the current data, parameters, and objective. Global importance requires broader evidence across observations and perturbations.
9. What comparison should precede a claim that an MLP is useful?
Compare it with a simple baseline under the same valid split and metrics, then examine repeated runs, uncertainty, failures, and held-out performance.
10. Why can two runs of the same neural-network code differ?
Weight initialization, mini-batch order, stochastic layers, and some numerical kernels introduce randomness. Record seeds and report variation rather than selecting only the best run.
Continue in Python
Optimization and MLP notebook
The downloadable notebook moves from a linear baseline to a small MLP on the same two-feature dataset.
Core lane
Run a reproducible baseline, inspect one autograd step, compare learning rates and hidden widths, then save a training trace and one evidence-based intervention.
Evidence artifact: training trace plus a short explanation of why one change helped, failed, or remained inconclusive.
Stretch lane
Implement a minimal two-layer network or one backpropagation step from basic tensor operations, then compare its gradients with autograd.
Glossary
Key terminology
- Tensor
- A multidimensional numerical array with a shape and data type.
- Parameter
- A value learned from data, such as a weight or bias.
- Logit
- An unrestricted model score before conversion to a probability.
- Activation function
- A nonlinear transformation applied between affine layers.
- Loss function
- The differentiable objective minimized during training.
- Gradient
- A local rate of change of the loss with respect to a parameter.
- Backpropagation
- Efficient application of the chain rule through a computational graph.
- Automatic differentiation
- Software that records operations and computes derivatives from them.
- Learning rate
- The scale applied to an optimizer's proposed parameter update.
- Generalization
- Performance on relevant data not used to fit the model.
References