GoKawiilRobert Triggs / Android Authority
AI is fast approaching a fork in the road. The spiraling cost of memory for both personal and cloud platforms, rising compute costs for training next-gen models, and high user demand for cloud platforms have made AI quite expensive to both provide and use. Then there’s the growing privacy concerns — just how much sensitive personal information are we giving away to chatbots?
The solution to all these problems increasingly looks like running models on your own devices. Whether that’s Gemini’s Nano models summarizing your emails on your phone or tweaking your LoRAs to generate images on your PC’s GPU, running AI locally is already a powerful tool that, in theory, can do even more without an internet connection.
However, this is easier said than done. Running your own large language models not only requires a large pool of expensive, fast RAM, but you also need an accelerator to crunch those numbers. While our smartphones have long supported NPUs for exactly this purpose, proprietary APIs and inconsistent software support have limited widespread adoption. Not to mention that flagship and mid-range phones have vastly different capabilities, meaning developers often have to maintain multiple code paths or fall back to slower CPU implementations anyway.
On-device AI is great, but it's a fractured space of proprietary APIs and hardware.
Instead, accelerating basic AI workloads on the CPU is an increasingly popular middle ground. It might not be anywhere near as fast or as capable as running GPT-OSS on your NVIDIA 5090, but targeting the CPU means the tool runs on essentially anything, without the development headaches of proprietary drivers and APIs. With the right building blocks deeper in the CPU, this doesn’t have to be as slow as it might sound at first.
Faster chips are just the start
Dhruv Bhutani / Android Authority
Mobile CPUs began moving toward more capable AI compute capabilities with Armv9 in 2021, which introduced SVE2. Rather than a fixed-width single instruction multiple data (SIMD) design, SVE2 uses a vector-length-agnostic model, allowing implementations to scale SIMD width from 128-bit up to 2048-bit depending on the hardware. In other words, it is a more flexible approach to doing fast math in parallel. It also added support for INT8 dot-product operations and enhanced support for low-precision arithmetic such as FP16, making it well-suited for modern quantized AI workloads.
However, the real shift came with SME and SME2. SME2 extends the CPU with a dedicated matrix execution mode, including a new matrix register and hardware support for GEMM-style operations. Unlike traditional vector execution, SME2 is designed specifically for the dense linear algebra patterns that dominate transformer and LLM workloads, significantly reducing memory traffic and increasing throughput for matrix-multiplication-heavy tasks.
... continue reading