A developer profiling Apple's M3 Neural Engine found an RTL performance bug that cuts DRAM weight-streaming speed from a normal 45-60 GB/s down to just 17-19 GB/s whenever a model's total weight size lands on an exact multiple of 1 MiB, affecting 7 of the 15 models in the ANEMLL project. By rerouting around the flawed speculative prefetch path in the kernel's DMA engine, the developer restored full throughput, more than doubling token generation speeds for Llama 3.2 1B and Qwen3-8B.
eiln.github.io
· 2026-09-10
A follow-up experiment tested four open models—DeepSeek V4 Flash, Inkling, Kimi K3, and Qwen3.8 A95B—by inserting the first 1% of GPT-5.5 Pro's reasoning trace into each model's own reasoning channel before letting it generate answers freely. Researchers then measured how much of GPT-5.5 Pro's visible answer text overlapped with each model's output. Qwen3.8 showed the largest jump, with overlap rising from 33.92% unprefilled to 54.50% with the GPT-5.5 Pro prefill, a 20.58 percentage-point increase, while other models showed much smaller shifts.
gist.github.com
· 2026-09-09
A benchmarking study tested several GGUF quantizations of the Qwen3.8 27B model against benchmarks including GPQA Diamond, IFBench, and Terminal-Bench 2.1. The 17GB Q4_K_M version performed on par with the full 55GB BF16 model on coding tasks and fits on a consumer 24GB GPU, while the smallest 6.2GB 1-bit version dropped to near-random accuracy on reasoning tasks, with longer reasoning chains making results worse.
quesma.com
· 2026-09-08
An independent experiment explored inverting the 'J space' Jacobian lens, normally used to decode what an LLM is about to say, to instead generate activation steering vectors directly from a handful of concept tokens. Using Qwen3-1.7B and Neuronpedia's published J lens, the method successfully steered simple behaviors like all-caps output or unusual speech patterns, benchmarked against existing steering vectors and an abliterated refusal-removed model.
darshanmakwana412.github.io
· 2026-09-06
A developer detailed a personal setup running local language models on an Apple M4 Pro Mac mini, using Qwen and Gemma models served through an inference tool called oMLX, connected across devices via Tailscale. The setup powers an agent backend called Hermes plus various chat and coding tools, and reportedly takes about 30 minutes to configure.
lws.io
· 2026-09-01
A new open-source tool called slotstream enables Apple Silicon Macs to run the 104GB Qwen3.8-Flash-Next language model even when the machine doesn't have enough RAM to hold it, by streaming weights from disk and dynamically sizing memory use, down to an 8.1GB floor. On a 48GB Mac it delivers about 12 tokens per second in warm decode with a roughly 3-second cold start, capping memory use at 32GB so the rest of the system stays usable. It requires macOS 14+, Apple Silicon, and about 110GB of free disk space for the one-time weight download.
github.com
· 2026-09-01
A user ran Alibaba's new Qwen3.8 27B model locally on a Mac Studio M3 Ultra for 10 days, using it for daily tasks like summarizing RSS feeds and renaming scanned PDFs, then formally benchmarked it against its predecessor. The Q4_K_M version generated at about 14 tokens per second versus 28.6 tokens/s for Qwen3.6 27B, but produced answers using roughly a third as many tokens, making total response time nearly equal. A 1-bit quantized version ran much faster at 27 tokens/s and stayed factually accurate but struggled to commit to definitive answers.
terminalbytes.com
· 2026-08-28
A hobbyist built CarWatch, a system that puts a Raspberry Pi 5 in a car running a 35-billion-parameter Qwen model locally, no internet or subscription required. It joins chat rooms as an AI agent, sending trip summaries, dashcam clips, and answers sourced from the car's 745-page owner's manual, and also supports hands-free voice queries via on-device speech recognition.
github.com
· 2026-08-25
GoKawiil