Tensors and Computational Linear Algebra

1. Tensor objects

Most mathematical introductions begin with scalars, vectors, and matrices. Deep-learning software extends this familiar sequence to arrays with any number of axes. PyTorch calls these arrays tensors. The same object type can hold raw observations, learned parameters, predictions, losses, and the intermediate activations produced inside a network.

A scalar contains one value and has no axes. A vector has one axis. A matrix has two axes. A higher-order tensor has three or more. These names describe organization, not meaning. A vector of length five might represent five measurements from one person, five class scores, or five time points. Interpretation begins only when we name the axis.

The term rank is potentially confusing. In tensor software, rank often means the number of axes. In linear algebra, matrix rank measures the number of linearly independent directions. This chapter uses tensor rank only when the context is explicit; otherwise, “number of axes” is clearer.

2. Shape is necessary but not sufficient

The shape of a tensor is the ordered list of axis sizes. If a tensor has shape \((n_1,\ldots,n_d)\), it stores

A tensor shaped \((2,3,4)\) therefore stores 24 values. This calculation determines whether a reshape is mechanically possible, but it does not tell us what the axes mean.

Consider a tensor shaped \((32,12,5)\). In a longitudinal study, the axes could mean participants, days, and measurements. In a language task, they could mean sentences, token positions, and embedding features. Both interpretations are compatible with the same shape. A reliable analysis records an axis contract: an ordered name for every axis, its unit, and its expected size.

Common conventions

Tabular models usually receive batch × features. Sequence models often receive batch × time × features, although some APIs use time × batch × features. Image models in PyTorch commonly use batch × channels × height × width. Conventions are useful, but they are not universal; inspect the specific model interface.

3. Creating tensors and reading their attributes

PyTorch can create tensors from Python data, NumPy arrays, random generators, or constant-filled constructors. For learning, direct creation from small visible values is preferable because every output can be checked by hand.

features = torch.tensor(
    [[18, 2, 7], [21, 1, 5]],
    dtype=torch.float32,
)
print(features.shape)   # torch.Size([2, 3])
print(features.dtype)   # torch.float32
print(features.device)  # cpu

Four properties deserve immediate attention:

1. Shape: Are the axes in the intended order?
2. Dtype: Are values represented appropriately?
3. Device: Are tensors that will interact stored on compatible devices?
4. Values: Does a small slice look plausible?

Floating-point types are standard for continuous inputs and model parameters. Integer types remain important for count data, indices, and class labels. Boolean tensors represent masks. Converting everything to floating point is not a universal solution because downstream operations may require the original semantic role.

Device placement is kept deliberately simple in this chapter. The required work runs on the CPU. The durable lesson is that device is an attribute to inspect and align, not a reason to move every introductory computation to an accelerator.

4. Indexing and slicing

Indexing chooses positions. PyTorch uses zero-based indexing, so index 0 refers to the first position and index 1 to the second. The colon means “all positions on this axis.”

x = torch.arange(12).reshape(3, 4)
x[1, :]    # second row, shape (4,)
x[:, 2]    # third column, shape (3,)
x[0:2, 1:3]  # two rows and two columns, shape (2, 2)
x[1, 2]    # one scalar, shape ()

An integer index removes the selected axis. A range slice preserves it. Thus x[1, :] has one axis, whereas x[1:2, :] remains a two-axis matrix with one row. This difference matters when a later function expects a batch axis.

Negative indices count from the end. In a tensor shaped batch × time × features, x[:, -1, :] selects the final time point for every batch member and retains every feature. Its shape is batch × features. This pattern is common in sequence prediction, but its meaning depends on the data being ordered correctly and free from future leakage.

5. Shape-changing operations

reshape, transpose, permute, squeeze, and unsqueeze all change how a tensor is addressed, but they answer different questions.

Reshape and flatten

A reshape preserves the number of values. A tensor of shape \((2,3,4)\) can become \((6,4)\), \((2,12)\), or \((24,)\). The condition is:

This condition is necessary but not sufficient for a meaningful transformation. Flattening participants and days into one axis may be useful for a model that treats participant-days as independent observations, but it also removes the structural distinction between people and time. The code cannot decide whether that loss of structure is acceptable.

Transpose and permute

A matrix transpose exchanges rows and columns. A general transpose exchanges two selected axes. permute specifies a complete new axis order. Image data illustrate the need: data read as batch × height × width × channels may need to become batch × channels × height × width for a PyTorch convolution.

Size-one axes

unsqueeze adds an axis of size one; squeeze removes size-one axes. These operations help represent a one-item batch, a single channel, or a broadcastable summary. Avoid an unrestricted squeeze() when a batch might contain exactly one observation, because it could remove the batch axis unexpectedly.

6. Reductions

A reduction summarizes values along one or more axes. A sum, mean, maximum, or norm can reduce an entire tensor to a scalar or remove selected axes. Interpretation requires answering two questions: which unit is combined, and which unit remains?

For a matrix \(X\in\mathbb{R}^{n\times p}\) containing observations by features, the mean of feature \(j\) is

In code, X.mean(dim=0) combines observations and retains features. The result has shape \((p,)\). By contrast, X.mean(dim=1) combines features and retains observations, producing shape \((n,)\).

keepdim=True retains a reduced axis with size one. If row means have shape \((n,1)\), subtracting them from an \((n,p)\) matrix broadcasts each row mean across that row's features. Retaining the axis makes the intended alignment visible.

7. Broadcasting

Broadcasting permits elementwise operations on differently shaped tensors without manually copying values. Shapes are aligned from the final axis toward the first. At every aligned position, the sizes must be equal or one of them must be 1. Missing leading axes behave as size 1.

Now compare \((2,3)+(2,)\). Right alignment compares 3 with 2, and neither size is 1. The operation is incompatible. Adding an explicit size-one axis can sometimes express the desired operation, but it should follow a semantic decision rather than trial and error.

8. Computational linear algebra

Elementwise arithmetic applies an operation to corresponding positions. If two matrices have the same shape, A * B multiplies matching entries and preserves that shape. Matrix multiplication performs a different operation.

If \(A\in\mathbb{R}^{m\times k}\) and \(B\in\mathbb{R}^{k\times n}\), then \(AB\in\mathbb{R}^{m\times n}\), with

The shared inner dimension \(k\) is combined. The outer dimensions \(m\) and \(n\) remain. This shape rule is the easiest first check for a proposed matrix product.

From dot products to neural layers

A dot product combines two equal-length vectors into one scalar. A matrix–vector product applies one linear transformation to a feature vector. A matrix–matrix product applies several transformations or transforms a batch at once. If a batch \(X\) has shape observations × input features and a weight matrix \(W\) has shape input features × outputs, then \(XW\) has shape observations × outputs.

This is the computational skeleton of linear regression and fully connected neural-network layers. Chapter 3 will add parameters and a loss; the present chapter establishes what must match before learning can begin.

Norms measure size

A norm summarizes the magnitude of a vector without retaining its direction. Two common choices are

The \(L_1\) norm treats every unit of absolute magnitude equally. The \(L_2\) norm gives proportionally more influence to large coordinates because they are squared before summation. For the vector \((3,4)\), the two norms are 7 and 5. Neither is universally better: the choice must match the quantity being summarized.

Norms recur throughout deep learning. A loss may summarize a prediction error, regularization may penalize parameter magnitude, and gradient clipping may limit a gradient norm. This chapter uses norms only as descriptive calculations; their role in learning is introduced later.

9. Batches and input contracts

A batch groups observations so the same operation can be applied efficiently. The first axis is often the batch axis, but “often” is not a contract. Sequence APIs and historical code sometimes place time first. A reliable workflow writes the expected order next to the tensor.

An input contract is not bureaucratic documentation. It is an executable debugging hypothesis. Assertions can check the number of axes, the final feature size, finite values, and allowed ranges. When an assertion fails, the message should name the violated meaning, not merely repeat a numeric shape.

def check_tensor(x, axis_names, expected_last_dim):
    assert x.ndim == len(axis_names)
    assert x.shape[-1] == expected_last_dim
    assert torch.isfinite(x).all()
    return {"shape": tuple(x.shape), "axes": axis_names}

10. Worked case: repeated measurements

Suppose eight participants are observed for six days, with four features measured each day. The natural tensor shape is participants × days × features, or \((8,6,4)\).

Question 1: one participant

data[0, :, :] selects the first participant, every day, and every feature. The result is days × features, shape \((6,4)\).

Question 2: the final day

data[:, -1, :] selects every participant, the last day, and every feature. The result is participants × features, shape \((8,4)\).

Question 3: feature means

data.mean(dim=(0,1)) combines participants and days while retaining features. The result has shape \((4,)\). It answers “what is the overall mean of each feature?” but ignores participant-level and temporal variation.

Question 4: feature offsets

Adding a vector shaped \((4,)\) broadcasts across participants and days because the final feature axis matches. A vector shaped \((6,)\) would not align with the final size-four axis; to apply day-specific offsets, the intended shape should be written explicitly, such as \((1,6,1)\).

Question 5: flattening

data.reshape(-1,4) produces \((48,4)\). This can be useful when each participant-day is treated as an observation. It also discards the visible distinction between participant and day. A later train/test split could then leak measurements from the same participant across sets. The tensor operation is valid, but the study design may not be.

11. Failure patterns and repairs

The repeated repair is interpretation before execution. Tiny, hand-checkable tensors are not childish examples; they are controlled experiments for verifying the meaning of a computation.

12. Conclusion

Tensors are the common representation through which deep-learning data and calculations pass. Their shapes organize values, but only a documented axis contract gives those shapes meaning. Indexing selects, reshaping regroups, transposition and permutation reorder, reductions summarize, broadcasting aligns, and matrix multiplication transforms. Each operation has both a mechanical rule and a substantive interpretation.

The practical outcome of this chapter is a habit: before running a model, write the observation unit, axis order, shape, dtype, device, expected final dimension, and one inspected slice. This habit will make automatic differentiation in Chapter 2 easier to understand because the values flowing through the computation graph will already have a clear structure.

Exercises and review

Glossary

Tensor

A multidimensional array whose values share a data type.

Scalar

A tensor with no axes; it contains one value.

Vector

A one-dimensional tensor.

Matrix

A two-dimensional tensor arranged by rows and columns.

Axis

One direction of variation in a tensor.

Shape

The ordered sizes of all tensor axes.

Dimension

A term often used for either an axis or the size of an axis; state the intended meaning.

Rank

In this chapter, the number of tensor axes; this differs from matrix rank.

Dtype

The numeric representation shared by a tensor's values.

Device

The processor and memory location holding a tensor.

Index

A position used to select a value or slice along an axis.

Slice

A selected range of positions from a tensor.

Reshape

A rearrangement of the same number of values into a new shape.

Transpose

An exchange of two axes; for a matrix, rows and columns are exchanged.

Permute

A reordering of three or more axes.

Reduction

An operation such as sum or mean that combines values along one or more axes.

Broadcasting

Automatic alignment of compatible shapes for an elementwise operation.

Elementwise operation

An operation applied to corresponding positions.

Matrix multiplication

A row-by-column operation that contracts a shared inner dimension.

Norm

A nonnegative measure of the size of a vector or tensor.

Batch

A group of observations processed together.

References

This chapter draws on the tensor and linear-algebra material taught in the 2023–2025 Advanced Data Science course and has been rewritten as a self-contained student resource. The following documentation provides stable technical references.

• PyTorch: Tensors
• PyTorch: Broadcasting semantics
• PyTorch: torch.reshape
• Dive into Deep Learning: Data Manipulation
• Dive into Deep Learning: Linear Algebra