GoKawiilReport details GPU, CPU and NPU benchmarks for running local AI models
New testing data compares how different hardware handles local large language models, measuring output in tokens per second. Nvidia's RTX 5090 generated 205 tokens per second on gpt-oss-20b, Apple's M4 chip produced about 24 tokens per second on a 7B model, and Nvidia's DGX Spark ranged from 38 tokens per second on Llama 3.1 8B down to 4.42 on its 70B version. Separate hands-on testing on a 2024 Asus Zenbook Duo showed Intel's OpenVINO toolkit offloading work to an NPU kept the laptop responsive, versus heavy CPU and memory strain when running the same model through Ollama.
GoKawiil's interpretation of the reporting above, not reported fact.
The comparisons suggest that memory bandwidth, not raw processing power, is the main bottleneck for running AI models locally, which could guide consumers and developers toward GPUs with faster VRAM or dedicated NPUs rather than relying on CPU power alone. This positions specialized hardware like Nvidia's DGX Spark or NPU-equipped laptops as potentially more practical for sustained local AI use, though performance still varies sharply with model size.
- Nvidia's RTX 5090 reached 205 tokens per second on a 20B parameter model, far outpacing other tested hardware.
- Larger models dramatically reduce speed, as shown by DGX Spark dropping from 38 to 4.42 tokens per second between an 8B and 70B model.
- Using a dedicated NPU via Intel's OpenVINO toolkit kept a laptop responsive, while standard CPU-based processing strained system resources.
NVIDIA GeForce RTX 5090 Graphics Card — If you're serious about running local AI models at usable speeds, the RTX 5090's massive VRAM and memory bandwidth are exactly what the article highlights as the key bottleneck for token generation. It's the card benchmarked hitting over 200 tokens per second on a 20B parameter model, far outpacing CPU or unified-memory setups. For anyone building a dedicated local AI rig, this is the GPU to beat.
See NVIDIA GeForce RTX 5090 Graphics Card on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: engadget.com — Luke James, 2026-11-09
Published there as: “What hardware does your PC need to run a local AI model?”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.