← Back to articles

Getting 50 GB/S Back from the Apple Neural Engine

2026-09-10 | original

read original get Apple Mac mini (M4) → more articles

Why This Matters

A developer reverse-engineering Apple's Neural Engine found an apparent hardware/firmware erratum on the M3 where weight-streaming DRAM throughput collapses from ~45-60 GB/s to ~17-19 GB/s whenever total weight size is an exact multiple of 1 MiB. Since model dimensions like 2048 are extremely common defaults, this silently halves or worse the on-device LLM performance for many models. Sidestepping the problem path roughly doubled to tripled token throughput on Llama 3.2 1B and Qwen3-8B.

Key Takeaways

An M3 ANE erratum throttles DRAM weight streaming to 17-19 GB/s when weight size is an exact multiple of 1 MiB, hitting 7 of ANEMLL's 15 models.
Avoiding the bad DMA prefetch path lifted Llama 3.2 1B from 10.0 to 24.3 tokens/s and Qwen3-8B from 1.36 to 2.97 tokens/s.
Common model dimensions (D=2048) trigger the bug while nearby values (D=2016) run ~2.6x faster, showing large free gains from alignment-aware tuning.

Worth a Look

Apple Mac mini (M4) — If this deep dive into Neural Engine weight streaming has you itching to run Llama or Qwen models locally, the Apple Mac mini with the M4 chip is a compact way to get Apple silicon's Neural Engine and unified memory on your desk. It's an inexpensive entry point for experimenting with ANEMLL and on-device LLM inference without a giant tower.

See Apple Mac mini (M4) on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.

Getting 50 GB/s Back Out of the ANE

Introduction

An RTL performance erratum in the Apple M3 Neural Engine throttles DRAM weight streaming throughput down to 17–19 GB/s from the nominal 45–60 GB/s, whenever the total weight size is an integer multiple of 1 MiB, which currently affects 7 of ANEMLL’s 15 models. Avoiding the problematic path in the kernel DMA engine's speculative prefetch ring increased Llama 3.2 1B token throughput from 10.0 to 24.3 tokens/s (DRAM usage from 24.7 to 60.0 GB/s), and Qwen3-8B from 1.36 to 2.97 tokens/s (DRAM usage from 22.4 to 48.7 GB/s).

Discovery

I was profiling the neural engine's DRAM weight streaming throughput (GB/s) for single token decode:

At \(N=4096\), I noticed that \(D=1536\) ran nearly 3× faster than \(D=2048\), the default used in Llama 3.2.

STATIC (pure KernelDMA) median µs per replica, N=4096:

D 576 768 1024 1280 1536 2048 rep a 150.4 196.8 238.1 293.8 310.9 997.6 rep b 157.8 190.2 250.1 288.3 326.8 995.0 rep c 148.7 189.6 249.4 275.2 316.5 995.4

Sweeping the D around the neighborhood of D = 2048:

Huh?

... continue reading

Explore topics: apple neural engine anemll llama 3.2 qwen3-8b apple m3