Week 13 · Hands-on foundations

Pretraining, Transfer, and Fine-Tuning

When should we freeze, adapt, or replace a pretrained representation?

The current Week 12 notebook demonstrates BERT, GPT-2, DistilBERT, vision transformers, sentiment, generation, and question answering. The newer LLM notebooks add objectives, feature extraction, prompting, and alignment. Those examples change quickly. The durable lesson is transfer: a model learns a representation on one task or dataset, then a smaller target task tests whether that representation transfers.

This chapter condenses materials taught in the 2023–2025 Advanced Data Science course into a smaller core path. Optional depth remains available in the companion notebook.

By the end

Distinguish pretraining, feature extraction, and fine-tuning.
Freeze and unfreeze parameters deliberately.
Compare a task head with full-model adaptation.
Document source-target mismatch and limitations.

Low floor

Use the explorer and three short Python examples before changing a longer model.

High ceiling

Unfreeze the final backbone layer, use a smaller learning rate, and compare three seeds against the frozen baseline.

Core ideas

Three concepts to keep

Pretraining objective

A model first learns patterns from a source task or large dataset. The learned representation reflects that objective and data; it is not a universal definition of meaning.

Feature extraction

A frozen backbone produces features while a small task head learns from target labels. This is efficient and often a strong low-data baseline.

Fine-tuning

Some or all pretrained parameters continue updating on target data. Fine-tuning can adapt a representation, but it can also overfit, forget useful structure, or amplify source-target mismatch.

Mathematical intuition

One relationship worth keeping

\[\theta^*=\arg\min_{\theta}\,L_{\mathrm{target}}(g_{\theta}(f_{\phi}(x)),y)\]

Feature extraction freezes backbone parameters φ and learns head parameters θ. Fine-tuning also updates some or all of φ.

Stretch: read the symbols slowly

Identify the input, learned quantity, output, and aggregation. Re-create the relationship with a tiny hand-checkable example before using a library layer.

Interactive explorer

Change one assumption at a time

Predict the direction of change before moving a control.

Target labels available 3Source-target shift 4Percent frozen 100

Current evidence—

Interpretation—

Change a control to inspect the relationship.

Hands-on path

A reusable five-step routine

1. Record source data and pretraining objective.
2. Define the target task, target population, and held-out evidence.
3. Start with frozen features and a small head.
4. Unfreeze only when the frozen baseline has a diagnosed limitation.
5. Compare gains, compute, instability, and target-domain failure slices.

Minimal Python

Predict, run, and explain

Each example is independent. Explain its output before copying it into a larger workflow.

Freeze a backbone

import torch
backbone=torch.nn.Linear(4,3)
for p in backbone.parameters(): p.requires_grad=False
print([p.requires_grad for p in backbone.parameters()])

output

Predict the result, then click Run.

Count trainable parameters

backbone, head = 15, 8
frozen_trainable = head
full_trainable = backbone + head
print(frozen_trainable, full_trainable)

output

Predict the result, then click Run.

Compare transfer evidence

frozen=[0.78,0.80,0.79]
fine=[0.79,0.85,0.74]
print(round(sum(frozen)/3,3), round(sum(fine)/3,3))

output

Predict the result, then click Run.

Failure checks

What can look correct while being wrong?

Pretrained means correct

What goes wrong: Source capability is assumed to transfer automatically.

Check: Test on target-domain held-out cases.

Fine-tune by default

What goes wrong: All parameters update despite few labels.

Check: Establish the frozen-feature baseline first.

Benchmark contamination

What goes wrong: Target test examples influenced pretraining or adaptation.

Check: Document data provenance and keep an independent evaluation set.

Check your understanding

Ten questions with standard answers

Answer in your own words before opening each panel.

1. When should we freeze, adapt, or replace a pretrained representation?

Standard answer: A strong answer connects the central idea to a visible computation and a held-out or shape-based check. The durable lesson is transfer: a model learns a representation on one task or dataset, then a smaller target task tests whether that representation transfers.

2. What does pretraining objective mean here?

Standard answer: A model first learns patterns from a source task or large dataset. The learned representation reflects that objective and data; it is not a universal definition of meaning.

3. Why is feature extraction useful?

Standard answer: A frozen backbone produces features while a small task head learns from target labels. This is efficient and often a strong low-data baseline.

4. How should you interpret fine-tuning?

Standard answer: Some or all pretrained parameters continue updating on target data. Fine-tuning can adapt a representation, but it can also overfit, forget useful structure, or amplify source-target mismatch.

5. What does the main equation clarify—and what does it not prove?

Standard answer: Feature extraction freezes backbone parameters φ and learns head parameters θ. Fine-tuning also updates some or all of φ. It does not by itself prove useful behavior on unseen intended-use cases.

6. What should change when you move the explorer controls?

Standard answer: The visible calculation and interpretation should change together. Predict the direction first, then use the result to correct your mental model.

7. What is the first hands-on check you should perform?

Standard answer: Record source data and pretraining objective.

8. How can you detect the failure called “Pretrained means correct”?

Standard answer: Test on target-domain held-out cases.

9. What evidence should be recorded before making a claim?

Standard answer: Record data and split assumptions, input/output shapes, settings, the baseline, held-out metrics, representative failures, and the decision supported by that evidence.

10. How does this week prepare the next topic?

Standard answer: Week 14 is reserved entirely for the final PBL presentations and course synthesis.

Terminology

Glossary

Pretraining: Learning representations on a source objective before a target task.
Backbone: Reusable representation-producing part of a model.
Task head: Small output component for a target prediction.
Feature extraction: Using a frozen backbone to produce target-task features.
Fine-tuning: Updating pretrained parameters on target data.
Freezing: Preventing selected parameters from receiving updates.
Domain shift: Difference between source and target data distributions.
Transfer: Reuse of learned knowledge across tasks or domains.
Catastrophic forgetting: Loss of useful prior behavior during adaptation.
Provenance: Traceable record of data, model, and transformation origins.

Go further

Key references

Next: Week 14 is reserved entirely for the final PBL presentations and course synthesis.