A community patch to llama.cpp (164 lines across two files) adds an Arm NEON inner kernel using the SDOT instruction for the tiled K-quant matmul routine, which previously lacked an Arm-specific path for formats like Q3_K, Q2_K and the IQ family. Benchmarks on Qwen2.5 1.5B across three Arm Neoverse server CPUs (N1, V1, V2) show prompt-processing speedups ranging from about 1.7x to 6.5x for low-bit quantized formats, with the largest gain on IQ3_M on Neoverse N1; formats that already had an Arm kernel, such as Q4_K_M and Q8_0, show no change, and token generation speed is unaffected.
github.com
· 2026-11-13
Huawei has released MACaQ, an internal quantizer that compresses text, image, video and audio models for local inference across llama.cpp, vLLM, Diffusers and PyTorch. The company says MACaQ can quantize a 122-billion-parameter model in 64 minutes using 32.5GB of device memory, and claims it achieves lower perplexity than rivals MR-GPTQ and DuQuant++ or matches their Llama-3-8B results in roughly a tenth of the time.
github.com
· 2026-11-11
A developer known for the nullprogram.com blog has released Prospero, an open-source coding harness built in C++ with a Dear ImGui interface, designed to connect AI models to developer tools for agentic coding tasks. The tool claims to use roughly two orders of magnitude less memory than existing harnesses, which the author says are typically built on resource-heavy web stacks. Prospero does not bundle any model provider, requiring users to supply their own backend such as llama.cpp, OpenRouter, or the Anthropic or OpenAI APIs.
nullprogram.com
· 2026-10-31
A new project called PS5LM runs llama.cpp on a jailbroken PS5's GPU, loading GGUF language models and serving them to other devices on the same network, such as laptops and phones. It exposes OpenAI-style and Anthropic-style endpoints, works with tools like Claude Code, and can also run decision models that answer TypeSafe's /v1/systemone API, the same interface used by the PC tool Ollaya. The console uses its Vulkan backend on its RDNA 2 GPU, with 11.4 GB of shared memory available for models and context, and needs no PC, cloud connection or account to operate.
github.com
· 2026-10-16
A Reddit user known as AdIllustrious7246 built a fork of PS5LM, a llama.cpp port for jailbroken PS5 consoles, that uses the PS5_Vulkan graphics driver to run AI language models on the console's GPU instead of its CPU. The update lets the PS5 run a 27-billion-parameter model at 20 tokens per second, and smaller models like Gemma 4 E4B and Llama 3.2 3B at 61 and 124 tokens per second respectively, far faster than the earlier CPU-only version that was limited to about 6.25 GiB of memory.
tomshardware.com
· 2026-10-16
A project called Janas-LLM has published benchmarks showing it can run large mixture-of-experts models, including a 48GB Qwen3-Next-80B model, on a 32GB laptop by streaming model 'experts' from an NVMe SSD rather than keeping everything in RAM. Tests on an Intel Core Ultra 9 185H laptop with its integrated Arc GPU showed Janas-LLM generating text at 24 tokens/sec versus llama.cpp's 2.8 tokens/sec on that model, with similar gains across a dozen other open models from Qwen and Gemma families. The developers released a benchmarking tool, janas-bench, and installation instructions for Linux x86-64 so others can reproduce results on their own hardware.
github.com
· 2026-10-12
A developer has published Janus, an open-source single Go binary that runs local .gguf language models on AMD, Intel or Nvidia GPUs via Vulkan, or on CPU, without requiring Python, Docker or Ollama. It exposes an OpenAI-compatible API alongside a built-in web UI with chat, tool-calling, memory and file-handling features, and supports hot-swapping models without restarting.
github.com
· 2026-10-01
Developer Hayder Tirmazi published a set of performance optimizations for llama.cpp's prompt lookup decoding (n-gram speculative drafting), claiming up to 42x faster drafting and up to 2.6x lower memory use. The work draws on techniques from Daniel Lemire and Martin Ankerl, and Lemire subsequently contributed a further patch that Tirmazi says boosts drafting speed by an additional 4.2x, for a combined speedup of up to 140x.
jadidbourbaki.github.io
· 2026-09-26
A GitHub project called jevper reimplements the documented 'System One' wire format used by the Jev interface, letting developers run typed questions (choice, score, noul) against any OpenAI-compatible model instead of the hosted TypeSafe API. It works with self-hosted llama.cpp servers or other duck-typed clients, requiring no dependency on typesafe-sdk or openai at runtime, and is installable via pip as a Python 3.10+ package.
github.com
· 2026-09-23
A technical blog post demonstrates how the buzzy AI concept known as 'Jev' can be replicated using an open-source Qwen3-0.6B model and roughly 25 lines of Python code. The example loads the model, prompts it to classify an email as legitimate, spam, or phishing, and extracts probability scores directly from the model's output logits rather than relying on an external API or specialized training pipeline.
nobodywho.ai
· 2026-09-23
Following a rushed 'Lite' GGUF release four days after Qwen 3.8 27B launched, the team has now finished fully optimized ShapeLearn quantizations and benchmarked them against both the Lite versions and rival quants. The full models push the quality-versus-speed tradeoff further, with all five new variants topping the performance frontier across six GPU test configurations.
byteshape.com
· 2026-09-18
Georgi Gerganov, creator of llama.cpp, addressed Nvidia's acquisition of Hugging Face, noting Nvidia engineers have contributed code and hardware to the project for over a year. He said llama.cpp and its ggml backend will remain hardware-agnostic and community-driven despite the new corporate ties.
twitter.com
· 2026-09-04