Week 5 · Hands-on foundations

Classification, Logits, and Softmax

How does a neural classifier turn features into competing class scores?

The current classification materials cover softmax regression, information theory, FashionMNIST, implementation from scratch, and concise PyTorch code. This chapter keeps the durable pathway from logits to probabilities to cross-entropy, using small arrays before a full image dataset.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Distinguish logits, probabilities, predicted classes, and labels.
Explain why softmax outputs sum to one.
Calculate cross-entropy for one labeled case.
Build a one-layer multiclass classifier.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Change the data overlap and explain what happens to probability confidence and cross-entropy.

Core ideas

Three concepts to keep

Logits

A classifier first produces one unrestricted score per class. Scores are comparative evidence; they are not probabilities and need not be positive or sum to one.

Softmax

Exponentiation makes transformed scores positive, and normalization makes them sum to one. Adding the same constant to every logit does not change the probabilities.

Cross-entropy

For a labeled example, cross-entropy penalizes the negative log probability assigned to the correct class. Confident wrong predictions receive a large loss.

Mathematical intuition

One relationship worth keeping

\[p_k=\frac{e^{z_k}}{\sum_j e^{z_j}},\qquad L=-\log p_y\]

The softmax probability p_k comes from all logits together. The loss then selects the probability of the observed class y.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Logit A 2Logit B 1Temperature 1

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Encode labels as class indices.
2. Produce one logit per class without applying softmax inside `CrossEntropyLoss`.
3. Fit on training data using the familiar optimization loop.
4. Convert logits to probabilities only for interpretation.
5. Inspect per-class errors instead of reporting accuracy alone.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Normalize three logits

import torch
logits = torch.tensor([2., 1., 0.])
probs = torch.softmax(logits, dim=0)
print([round(float(v), 3) for v in probs])

output

Predict the result, then click Run.

Choose the predicted class

scores = [0.2, 1.4, -0.3]
print(max(range(len(scores)), key=scores.__getitem__))

output

Predict the result, then click Run.

Calculate one cross-entropy

import math
correct_probability = 0.7
print(round(-math.log(correct_probability), 3))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Double softmax

What goes wrong: Probabilities are passed into a loss that expects logits.

Check: Check the loss API and model's last layer.

Wrong class axis

What goes wrong: Softmax normalizes across observations rather than classes.

Check: Name the batch and class dimensions.

Imbalanced accuracy

What goes wrong: A majority class dominates the score.

Check: Inspect class counts and the confusion matrix.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. How does a neural classifier turn features into competing class scores?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. This chapter keeps the durable pathway from logits to probabilities to cross-entropy, using small arrays before a full image dataset.

2. What does logits mean here?

Standard answer: A classifier first produces one unrestricted score per class. Scores are comparative evidence; they are not probabilities and need not be positive or sum to one.

3. Why is softmax useful?

Standard answer: Exponentiation makes transformed scores positive, and normalization makes them sum to one. Adding the same constant to every logit does not change the probabilities.

4. How should you interpret cross-entropy?

Standard answer: For a labeled example, cross-entropy penalizes the negative log probability assigned to the correct class. Confident wrong predictions receive a large loss.

5. What does the main equation clarify—and what does it not prove?

Standard answer: The softmax probability p_k comes from all logits together. The loss then selects the probability of the observed class y. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Encode labels as class indices.

8. How can you detect the failure called “Double softmax”?

Standard answer: Check the loss API and model's last layer.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 6 moves from model mechanics to the complete classification workflow: splitting, thresholds, confusion matrices, and error trade-offs.

Terminology

Glossary

Classification: Prediction of a discrete category.
Class: One possible target category.
Label: Observed class assigned to an example.
Logit: Unnormalized model score for a class.
Softmax: Normalization converting logits into a probability distribution.
Probability: Nonnegative value representing relative class confidence.
Cross-entropy: Loss based on the predicted probability of the observed class.
Argmax: Index of the largest score or probability.
Multiclass: Task with more than two mutually exclusive classes.
Decision boundary: Input locations where the predicted class changes.

Go further

Key references

Next: Week 6 moves from model mechanics to the complete classification workflow: splitting, thresholds, confusion matrices, and error trade-offs.