A new project called PS5LM runs llama.cpp on a jailbroken PS5's GPU, loading GGUF language models and serving them to other devices on the same network, such as laptops and phones. It exposes OpenAI-style and Anthropic-style endpoints, works with tools like Claude Code, and can also run decision models that answer TypeSafe's /v1/systemone API, the same interface used by the PC tool Ollaya. The console uses its Vulkan backend on its RDNA 2 GPU, with 11.4 GB of shared memory available for models and context, and needs no PC, cloud connection or account to operate.
github.com
· 2026-10-16
A Reddit user known as AdIllustrious7246 built a fork of PS5LM, a llama.cpp port for jailbroken PS5 consoles, that uses the PS5_Vulkan graphics driver to run AI language models on the console's GPU instead of its CPU. The update lets the PS5 run a 27-billion-parameter model at 20 tokens per second, and smaller models like Gemma 4 E4B and Llama 3.2 3B at 61 and 124 tokens per second respectively, far faster than the earlier CPU-only version that was limited to about 6.25 GiB of memory.
tomshardware.com
· 2026-10-16
A project called Janas-LLM has published benchmarks showing it can run large mixture-of-experts models, including a 48GB Qwen3-Next-80B model, on a 32GB laptop by streaming model 'experts' from an NVMe SSD rather than keeping everything in RAM. Tests on an Intel Core Ultra 9 185H laptop with its integrated Arc GPU showed Janas-LLM generating text at 24 tokens/sec versus llama.cpp's 2.8 tokens/sec on that model, with similar gains across a dozen other open models from Qwen and Gemma families. The developers released a benchmarking tool, janas-bench, and installation instructions for Linux x86-64 so others can reproduce results on their own hardware.
github.com
· 2026-10-12
Meta unveiled two new AI models, Muse Wonder and Muse Wonder Full, which the company says deliver frontier-level capabilities. The models will be accessible via an API and through Muse Code, a coding tool, and the top version will power features across Meta's platforms including its Muse personal assistant. Meta says the largest model outperforms Anthropic's Claude Fable 5.1, though it did not disclose training or architecture details.
wired.com
· 2026-10-12
A developer has published Janus, an open-source single Go binary that runs local .gguf language models on AMD, Intel or Nvidia GPUs via Vulkan, or on CPU, without requiring Python, Docker or Ollama. It exposes an OpenAI-compatible API alongside a built-in web UI with chat, tool-calling, memory and file-handling features, and supports hot-swapping models without restarting.
github.com
· 2026-10-01
Developer Hayder Tirmazi published a set of performance optimizations for llama.cpp's prompt lookup decoding (n-gram speculative drafting), claiming up to 42x faster drafting and up to 2.6x lower memory use. The work draws on techniques from Daniel Lemire and Martin Ankerl, and Lemire subsequently contributed a further patch that Tirmazi says boosts drafting speed by an additional 4.2x, for a combined speedup of up to 140x.
jadidbourbaki.github.io
· 2026-09-26
A GitHub project called jevper reimplements the documented 'System One' wire format used by the Jev interface, letting developers run typed questions (choice, score, noul) against any OpenAI-compatible model instead of the hosted TypeSafe API. It works with self-hosted llama.cpp servers or other duck-typed clients, requiring no dependency on typesafe-sdk or openai at runtime, and is installable via pip as a Python 3.10+ package.
github.com
· 2026-09-23
A technical blog post demonstrates how the buzzy AI concept known as 'Jev' can be replicated using an open-source Qwen3-0.6B model and roughly 25 lines of Python code. The example loads the model, prompts it to classify an email as legitimate, spam, or phishing, and extracts probability scores directly from the model's output logits rather than relying on an external API or specialized training pipeline.
nobodywho.ai
· 2026-09-23
An AI researcher briefed members of Congress and staff on the current state of open-weight versus open-source language models, framed within U.S.-China competition. The briefing distinguished closed API-only models like GPT-4 and Claude from open-weight models such as Meta's Llama, Alibaba's Qwen and DeepSeek, and from fully open-source models like the Allen Institute's Olmo, noting Chinese firms have led in open-weight releases since around April 2025.
interconnects.ai
· 2026-09-22
Following a rushed 'Lite' GGUF release four days after Qwen 3.8 27B launched, the team has now finished fully optimized ShapeLearn quantizations and benchmarked them against both the Lite versions and rival quants. The full models push the quality-versus-speed tradeoff further, with all five new variants topping the performance frontier across six GPU test configurations.
byteshape.com
· 2026-09-18
A developer profiling Apple's M3 Neural Engine found an RTL performance bug that cuts DRAM weight-streaming speed from a normal 45-60 GB/s down to just 17-19 GB/s whenever a model's total weight size lands on an exact multiple of 1 MiB, affecting 7 of the 15 models in the ANEMLL project. By rerouting around the flawed speculative prefetch path in the kernel's DMA engine, the developer restored full throughput, more than doubling token generation speeds for Llama 3.2 1B and Qwen3-8B.
eiln.github.io
· 2026-09-10
Georgi Gerganov, creator of llama.cpp, addressed Nvidia's acquisition of Hugging Face, noting Nvidia engineers have contributed code and hardware to the project for over a year. He said llama.cpp and its ggml backend will remain hardware-agnostic and community-driven despite the new corporate ties.
twitter.com
· 2026-09-04
GoKawiil