GoKawiilTensor is the might
Every good abstraction solves a problem, and this post will cover everything I know so far about a brilliant math abstraction - tensors.
Neural networks, from a simple 2-layer MLP to GPT-5, all boil down to the same thing: floating-point numbers flowing through a graph of operations. This post builds a complete, accelerated tensor library from scratch in C. It is heavily inspired by Bellard’s libnc, which unfortunately has not been open-sourced yet.
A tensor is nothing but a flat array of numbers, plus some metadata telling you how to interpret those numbers as a multi-dimensional object. We all learned that 2D arrays can be better represented as 1D array plus a number of rows/columns - this is essentially what a tensor is.
But going beyond two dimensions - we might need some other metadata, such as a generalised shape:
float data [ 32 * 3 * 28 * 28 ]; // 32 images, 3 channels, 28x28 pixels int shape [ 4 ] = { 32 , 3 , 28 , 28 }; // shape of the tensor int ndim = 4 ; // number of dimensions
Having a shape we can figure out that an element at position [n,c,h,w] in a 4D tensor lives at offset data[n*(3*28*28)+c*(28*28)+h*28+w] , if we keep our tensor in a row-major C-order format (often the default in most tensor libraries today).
Now, calculating each time an index of an element like this is inefficient, so we can precompute the strides once the shape is known. Strides tell how many elements to skip if we want to advance for one element in the given dimension. We could also group all shape-related fields together:
struct ut_shape { int ndim ; // number of dimensions int nelem ; // number of elements int shape [ 4 ]; // shape of the tensor int strides [ 4 ]; // strides for each dimension }; struct ut_tensor { struct ut_shape shape ; // shape of the tensor float * data ; // pointer to the data ... // more fields to come later };
We can add some helpers to create a shape and get flat index from a multi-dimensional index:
... continue reading