PyTorch: Tensors & Autograd
Read the graph the engine built
Every tracked operation leaves a node behind, and backward() walks those nodes in reverse. This is practice at predicting both halves — what got recorded, and the exact number that comes back — from shapes and dtypes through to a gradient worked by hand, finishing with an engine of your own.
- Modules
- 4
- Drills
- 9
- Build steps
- 7
- Time
- ~4h
Walk out able to
Say what shape a tensor expression has, why a gradient is None, and what number backward() will put in any tensor of a small graph.
What you work through
What a tensor carries
Shape, dtype, and whether anything is watching.
Shapes that line up
Rearranging calls, batched @, the silent broadcast.
What the engine records
grad_fn, leaves, and the five reasons for None.
Walk the graph backwards
Gradients by hand, then an engine that agrees.
The pitch
What you practise, and what you leave with
autodiff.py: a reverse-mode engine over single numbers, run on a stated graph and checked digit for digit against the same graph in torch.
You will practise
- Dtypes, shapes and batched @
- Which lines built graph, and why None
- What the gradient buffer holds
- Gradients by hand, broadcast ones too
Afterwards you can
- Say the shape before running the line
- Diagnose a None gradient from the code
- Compute a small graph’s gradients by hand
- Write a working reverse-mode engine
Modules
4 modules, 28 items
Lessons explain one idea. Drills repeat it until it sticks. Build steps make something that exists afterwards.
- 1
What a tensor carries
~43 minName the three labels on any tensor, and predict the dtype a construction lands on.
- LessonWhy an array library keeps a diary5 min
- LessonRead the three labels a tensor carries9 min
- LessonPredict the dtype a line lands on5 min
- DrillSay the dtype before it bites10 min
- Build stepMake a tensor and read it back7 min
- Build stepTotal two columns, then divide7 min
- 2
Shapes that line up
~1hSay what shape any rearranging call or any matrix multiply produces — or that it raises.
- LessonGive a tensor the axis it needs10 min
- DrillLine two shapes up, or make them12 min
- LessonTwo spellings for the same rearrange6 min
- DrillSay the shape after the call12 min
- LessonWhat the axes in front of a matmul do6 min
- DrillSay the shape a matrix multiply produces12 min
- Build stepFind the broadcast that did not raise7 min
- Build stepGive it the axis it was missing7 min
- 3
What the engine records
~1hSay which lines built graph, where a gradient is kept, and what the buffer holds.
- LessonWatch the engine write it down8 min
- LessonFind out where a gradient is kept8 min
- LessonSwitch the recording off on purpose6 min
- DrillSpot the line that built no graph10 min
- DrillSay why the gradient is None12 min
- DrillSay what the gradient buffer holds10 min
- Build stepBreak the second backward pass12 min
- 4
Walk the graph backwards
~1hCompute a small graph’s gradients by hand, then build the engine that does it.
- LessonWalk a graph backwards by hand11 min
- DrillCompute it before the engine does14 min
- LessonA gradient wears its own shape6 min
- DrillSum a gradient back to its shape12 min
- LessonWhy backward() wants one number5 min
- Build stepBuild the engine that does it for you18 min
- Build stepMake it agree with torch, digit by digit14 min
More in AI & machine learning
Keep going
Say the gradient before backward() does.
The first item is free. ~4h of focused work, at your own pace.
Intermediate~4h4 modules