In defense of shallow technical knowledge

Whenever a new piece of technology comes out (these days, mostly AI) I go to some effort to understand it. Usually I end up writing a post about it, so I can be confident that I do understand. What’s the point of doing this? Obviously my explainers about diffusion models are shallow: certainly they aren’t detailed enough to do useful research on diffusion models. What’s the point, then? In my view, good engineering requires having reliable shallow intuitions about how things work. You don’t need a full understanding of how things work, or even a good enough understanding to work usefully in that area of the stack. But it’s still useful to try to minimize the number of technologies in your stack that are purely black boxes. Database indexes Let me give an example that isn’t about AI. If you’ve done any kind of web development, you rely heavily on database indexes. For some engineers, these are black boxes. All they know is what they do: specifically, that if you set up the right kin ... Read full article.