GoKawiilFigure 1. the program this whole series is about.
You have typed something like this a thousand times. This series exists so that, by its end, you know everything these lines do. All of it: the Python they touch, the C++ they land in, the graph they record, the kernels they choose, the memory they use, and the two clocks they run on. Each of those words gets a plain meaning on its floor below.
This is Part 0, the map. First we go down through all the layers once, fast. Then we draw the territory. Then twelve ideas that make the rest of the codebase predictable. Then how this series works, and how to read it. Nothing here gets its full story. Everything here gets a place, and every full story has a numbered part waiting for it.
One promise before we start. Every measured number in this series comes from a small script you can run yourself, linked right where the number appears. I measured these on an Apple M3 Max laptop with torch 2.11.0 [1]. Your numbers will differ. The pattern they make will not.
PyTorch is deep. Between your keyboard and the chip there are eight levels. I will call them floors, and this meter shows all of them. It returns through the whole series, so you always know how deep you are. Figure 2. the depth meter. the orange dot marks where you are.
The fastest way to learn a building is to go down through it once without stopping. That is this section.
floor 2 of 8 python full story: Part 5, The Machinery
torch.randn looks like a Python function. Ask Python what it actually is:
>>> type ( torch . randn ) <class ' builtin_function_or_method ' >
Python gives that type only to functions written in compiled code. Compiled code means: code that was translated to machine instructions before you ever installed it, so there is no Python body inside it to read, and no line for your debugger to stop on.
... continue reading