Local LLM inference – impressive but too hard to work with

Local LLM inference Amir Zohrenejad · Follow 4 min read · Just now Just now -- Listen Share Tremendous progress, but not ready for production I stumbled into local inference during a side quest. The main mission was to build a text-to-SQL product. Curiosity hijacked the roadmap: could I cram an entire GenBI stack into the browser? The prototype never shipped. But I did fall into the local inference rabbit hole. Though “AI inference” is not a listed feature on my laptop spec sheet — through the magic of open source software it can now run powerful LLMs in its browser tabs for free. It’s impressive. Just not quite production-ready as a developer platform. Why bother with local compute? From mainframes to PCs to the cloud, compute has swung between centralization and edge. Now it’s drifting back toward the edge — at least if you squint through the hype. But most users don’t actually care where computation happens. They want it to be fast, and they want it to be cheap. For example: F ... Read full article.