Odds are the PC in your office today isn’t ready to run AI large language models (LLMs).
Today, most users interact with LLMs via an online, browser-based interface. The more technically inclined might use an application programming interface or command line interface. In either case, the queries are sent to a data center, where the model is hosted and run. It works well, until it doesn’t; a data-center outage can take a model offline for hours. Plus, some users might be unwilling to send personal data to an anonymous entity.
Running a model locally on your computer could offer significant benefits: lower latency, better understanding of your personal needs, and the privacy that comes with keeping your data on your own machine.
However, for the average laptop that’s over a year old, the number of useful AI models you can run locally on your PC is close to zero. This laptop might have a four- to eight-core processor (CPU), no dedicated graphics chip (GPU) or neural-processing unit (NPU), and 16 gigabytes of RAM, leaving it underpowered for LLMs.
Even new, high-end PC laptops, which often include an NPU and a GPU, can struggle. The largest AI models have over a trillion parameters, which requires memory in the hundreds of gigabytes. Smaller versions of these models are available, even prolific, but they often lack the intelligence of larger models, which only dedicated AI data centers can handle.
The situation is even worse when other AI features aimed at making the model more capable are considered. Small language models (SLMs) that run on local hardware either scale back these features or omit them entirely. Image and video generation are difficult to run locally on laptops, too, and until recently they were reserved for high-end tower desktop PCs.
That’s a problem for AI adoption.
To make running AI models locally possible, the hardware found inside laptops and the software that runs on it will need an upgrade. This is the beginning of a shift in laptop design that will give engineers the opportunity to abandon the last vestiges of the past and reinvent the PC from the ground up.
NPUs enter the chat
The most obvious way to boost a PC’s AI performance is to place a powerful NPU alongside the CPU.
... continue reading