PythonPyTorchAdvanced~4h

PyTorch: Training

Make the loop actually update the model

A training loop is five calls, and getting one of them in the wrong place costs you a day — because nothing raises, nothing warns, and the loss simply sits there. This is practice at writing the loop, and at the part no course teaches: reading a failing run and naming the cause rather than changing the learning rate.

Start this projectSee the modules
Modules
4
Drills
10
Build steps
3
Time
~4h

Walk out able to

Write a training step that updates the model, and diagnose one that does not — from the symptom in the log and the line in the code.

What you drill

01

Batches and steps

How many, how big the last one, how often you step.

02

The five calls

What each one reads, and what a misplaced one does.

03

Optimisers

The update by hand, and the list it is applied to.

04

When it will not learn

Four symptoms, and the cause each one points at.

The pitch

What you practise, and what you leave with

A training script that is broken four ways at once, repaired until it learns — plus the account of which symptom named which fault.

You will practise

  • Batches, remainders and optimiser steps
  • What a misplaced call in the loop does
  • SGD and momentum, run by hand
  • Symptom to cause, on a failing run

Afterwards you can

  • Write a loop that updates the model
  • Say what the forgotten call cost you
  • Name a failure from its log, not by guessing
  • Know which parameters never move, and why

Modules

4 modules, 25 items

Lessons explain one idea. Drills repeat it until it sticks. Build steps make something that exists afterwards.

  1. 1

    Feed it in batches

    ~45 min

    Say how many batches an epoch runs, how big the last one is, and how often you step.

    • LessonWhy a training loop goes wrong5 min
    • LessonCount the batches before the loop8 min
    • LessonSay what the short batch costs you6 min
    • LessonCount the steps a run will perform6 min
    • DrillSay how many batches, and what is left8 min
    • Build stepBuild a loader and count what it yields12 min
  2. 2

    The five calls

    ~52 min

    Write a training step that updates the model, and read one that does not.

    • LessonWrite a step that updates the model9 min
    • LessonSay why gradients accumulate9 min
    • DrillSay what this training loop does12 min
    • DrillSay where the parameter lands10 min
    • Build stepDelete the clear and watch what it cost12 min
  3. 3

    Optimisers

    ~49 min

    Run an update by hand, and say which parameters your optimiser will never move.

    • LessonSee what the optimiser kept8 min
    • DrillSay what the optimiser owns10 min
    • LessonMove a parameter by hand9 min
    • DrillRun the optimiser's recursion12 min
    • Build stepTurn on momentum and re-run10 min
  4. 4

    Debugging a model that won't learn

    ~2h

    Name the cause from the symptom and the code, instead of changing things at random.

    • LessonRead the symptom before the code9 min
    • LessonRun the two cheap checks first9 min
    • DrillName the cause from the symptom14 min
    • DrillSay which line stopped the gradient10 min
    • DrillSay which call was forgotten10 min
    • DrillSay what keeps the graph alive10 min
    • DrillFind the call that is spelled wrong9 min
    • LessonFind the four faults from the log10 min
    • LessonRepair a run that will not learn15 min

More in AI & machine learning

Keep going

All AI & machine learning projects

RAG & Embeddings

Retrieval is arithmetic. Do the arithmetic.

Intermediate~4h5 modules

LLM Fundamentals

Tokens are countable. A window is arithmetic.

Intermediate~4h5 modules

PyTorch: Tensors & Autograd

Say the gradient before backward() does.

Intermediate~4h4 modules

Write a loop that learns, and fix one that does not.

The first item is free. ~4h of focused work, at your own pace.

Start PyTorch: TrainingAll projects

Advanced~4h4 modules