New NEON kernel speeds up llama.cpp prompt processing on Arm CPUs up to 6.5x
A community patch to llama.cpp (164 lines across two files) adds an Arm NEON inner kernel using the SDOT instruction for the tiled K-quant matmul routine, which previously lacked an Arm-specific path for formats like Q3_K, Q2_K and the IQ family. Benchmarks on Qwen2.5 1.5B across three Arm Neoverse server CPUs (N1, V1, V2) show prompt-processing speedups ranging from about 1.7x to 6.5x for low-bit quantized formats, with the largest gain on IQ3_M on Neoverse N1; formats that already had an Arm kernel, such as Q4_K_M and Q8_0, show no change, and token generation speed is unaffected.
GoKawiil's interpretation of the reporting above, not reported fact.
The fix addresses a gap introduced when the tiled K-quant matmul routine was added in September 2026 with optimized inner kernels only for x86 instruction sets (AVX-512 VNNI, AVX2, AVX), leaving Arm systems falling back to a slower unpacking path for several low-bit quantized formats. Closing that gap could make Arm-based servers and devices substantially more competitive for running quantized large language models, particularly for memory-constrained or power-efficient deployments that favor low-bit quantization. The benchmarks suggest Arm hardware can now approach parity with x86 for prompt processing in these formats, though real-world impact will depend on adoption into mainline llama.cpp.
- A new NEON SDOT kernel fixes a missing Arm optimization path in llama.cpp's tiled K-quant matmul routine.
- Prompt-processing speed for low-bit quantized formats (IQ3_M, IQ3_XS, IQ2_M, Q3_K, Q2_K) improved by 1.7x to 6.5x on Arm Neoverse CPUs.
- Formats that already had Arm kernels and generation speed overall remain unchanged by this patch.
Raspberry Pi 5 (8GB) — If you're experimenting with llama.cpp on Arm CPUs, the Raspberry Pi 5 is an affordable way to test NEON-optimized inference improvements like this one firsthand. Its Arm Cortex cores let you benchmark quantized model speedups right on your desk without needing server-grade hardware.
See Raspberry Pi 5 (8GB) on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.Source: github.com, 2026-11-13
Published there as: “Llama.cpp: 6.5x faster prompt processing on Arm with a NEON kernel”
Read the original report → The summary and analysis above are GoKawiil's own, written from reporting by the source above. Facts and quotes belong to the original publisher.