Chapter 1 · Foundations
How can shape, axes, and operations preserve—or quietly change—the meaning of data?
Deep-learning systems represent observations, parameters, and intermediate calculations as tensors. Before studying gradients or models, you need to read a tensor as structured data: what each axis means, which operations preserve that meaning, and which operations can silently change it.
Use small, visible tensors. Predict each output shape before running code, then connect the result to observations, features, channels, or time.
Trace a batch × time × features tensor through slicing, permutation, reduction, and a linear transformation while documenting every axis.
Tensor language
A tensor is a multidimensional array. Its rank in this chapter is the number of axes, and its shape lists the size of each axis in order. A scalar has shape (); a vector might have shape (5,); a matrix might have shape (32, 5). Higher-order tensors let us describe images, sequences, and batches without changing the underlying idea.
| Structure | Example shape | Possible meaning |
|---|---|---|
| Scalar | () | One loss value |
| Vector | (5,) | Five features for one observation |
| Matrix | (32, 5) | Thirty-two observations by five features |
| Three axes | (32, 12, 5) | Batch by time by features |
| Four axes | (32, 3, 28, 28) | Batch by channels by height by width |
A matrix \(X\in\mathbb{R}^{n\times p}\) contains \(n\) rows and \(p\) columns. In a typical tabular dataset, \(n\) is the number of observations and \(p\) is the number of features. For a tensor with shape \((n_1,\ldots,n_d)\), the number of stored values is:
Interactive explorer
Choose a data type and adjust its dimensions. The diagram and interpretation update together.
Indexing and slicing
PyTorch follows zero-based indexing. In a matrix x, x[1, :] selects the second row and every column; x[:, 0] selects every row and the first column. An integer index removes an axis, while a range such as 1:2 preserves it.
Ellipsis (...) stands for the axes not written explicitly. Negative indices count backward from the end, so -1 selects the final position.
Shape operations
Groups the same sequence of values into new axis sizes. The total number of values must remain constant.
Exchanges two axes. For a matrix, it changes rows × columns into columns × rows.
Reorders several axes, such as converting batch × height × width × channels into batch × channels × height × width.
Some shape operations can return a view that shares storage with the original tensor; others may require a copy. For this chapter, use reshape when you need a particular shape and do not build conclusions around storage sharing. Memory layout and contiguity are deferred.
Reductions
A reduction combines several values. On a matrix shaped observations × features, x.mean(dim=0) produces one mean per feature, while x.mean(dim=1) produces one mean per observation. Omitting dim combines every value into one scalar.
keepdim mattersx.mean(dim=1, keepdim=True) retains a size-one axis. The result can be subtracted from the original matrix to center every row with an explicit broadcast-compatible shape.
Broadcasting
For elementwise operations, PyTorch compares shapes from right to left. Two axis sizes are compatible when they are equal or one of them is 1. Missing leading axes are treated as size 1. Broadcasting avoids manual copying, but it cannot decide whether the alignment is meaningful.
Computational linear algebra
If \(A\in\mathbb{R}^{m\times k}\) and \(B\in\mathbb{R}^{k\times n}\), then \(AB\in\mathbb{R}^{m\times n}\). The shared inner size \(k\) is combined; the two outer sizes remain.
A dot product combines two vectors into a scalar. A matrix–vector product transforms one feature vector. A matrix–matrix product transforms a batch or composes linear maps. These operations will become the computational core of linear regression and neural-network layers.
The \(L_1\) norm adds absolute values, while the Euclidean or \(L_2\) norm is the square root of the sum of squares:
For x = torch.tensor([3., 4.]), torch.linalg.vector_norm(x, ord=1) is 7 and torch.linalg.vector_norm(x) is 5. Norms later help quantify errors, parameter size, and changes in gradients.
A reusable debugging habit
meaning: batch × time × features
expected shape: (32, 12, 5)
dtype: torch.float32
device: cpu
model expects: 5 features at the final axis
checked slice: x[0, -1, :]Minimal Python
Each example is independent and bounded. Predict the shape and output before selecting Run.
import torch
features = torch.tensor(
[[18, 2, 7], [21, 1, 5]],
dtype=torch.float32,
)
print(f"shape={tuple(features.shape)}")
print(f"dtype={features.dtype}, device={features.device}")Click Run to see the results.
import torch
x = torch.arange(12).reshape(3, 4)
second_observation = x[1, :]
print(second_observation.tolist())
print(f"shape={tuple(second_observation.shape)}")Click Run to see the results.
import torch
x = torch.arange(24).reshape(2, 3, 4)
y = x.reshape(6, 4)
print(f"{tuple(x.shape)} -> {tuple(y.shape)}")
print(f"same values={torch.equal(x.flatten(), y.flatten())}")Click Run to see the results.
import torch
x = torch.tensor([[1., 2., 3.], [4., 5., 6.]])
column_means = x.mean(dim=0)
print(column_means.tolist())
print(f"shape={tuple(column_means.shape)}")Click Run to see the results.
import torch
x = torch.tensor([[1., 2.], [3., 4.]])
shift = torch.tensor([10., 20.])
print((x + shift).tolist())Click Run to see the results.
import torch
A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[2., 0.], [1., 2.]])
print("A * B =", (A * B).tolist())
print("A @ B =", (A @ B).tolist())Click Run to see the results.
Common mistakes
Symptom: a model receives features × observations.
Check: inspect one row and state what it represents.
Symptom: the element count matches, but unrelated values are grouped as one observation.
Check: name the new axes before reshaping.
Symptom: an operation succeeds along an unintended axis.
Check: align shapes from the right on paper.
Symptom: one mean per observation is reported as one mean per feature.
Check: describe what disappears and what remains.
Symptom: an operation fails or silently represents data inappropriately.
Check: print dtype and preserve integer class labels where required.
* versus @Symptom: a plausible-shaped result implements the wrong calculation.
Check: say “corresponding entries” or “row by column” before coding.
Summary
Next: Chapter 2 adds derivatives and automatic differentiation to the tensor operations established here.
Review
One plausible interpretation is 32 observations by 5 features. Shape alone does not establish that meaning; the data contract must name the axes.
They have zero, one, two, and three axes respectively. These names describe structure, not the substantive meaning of the values.
Shape records axis sizes, but not whether an axis means people, features, channels, time steps, or something else.
It keeps every observation, selects the final time step, and keeps every feature. The result is shaped batch × features.
The original and requested shapes must contain the same total number of values. Semantic validity also requires that the new axes have an intended meaning.
It retains the reduced axis with size one, which can make later broadcasting explicit and less error-prone.
Yes. Alignment starts from the right, so the size-3 vector matches the final size-3 axis of the matrix and is reused across four rows.
`A * B` multiplies corresponding entries and normally preserves shape. `A @ B` performs row-by-column multiplication and requires compatible inner dimensions.
Labels and model inputs play different roles. Many classification losses expect integer class indices even though model inputs and parameters use floating-point values.
Name the axes, inspect the shape, verify the dtype and device, and inspect a small slice or range for plausible values. Also compare the observed shape with the model's declared input contract.
Terminology
Sources