An independent developer has released a project that builds a transformer inference engine from scratch in C++17 and CUDA, documented through over six hours of video and an 80-plus page PDF of handwritten notes. The system currently loads Llama 3.2 1B Instruct weights directly from safetensors files, runs a custom Unicode-aware BPE tokenizer, and executes full dense causal prefill across all 16 layers including RoPE, grouped-query attention, SwiGLU MLP, and RMSNorm kernels to produce next-token predictions via GPU argmax.
github.com
· 2026-11-09
Tom's Hardware has released an explainer breaking down core terminology behind large language models, covering concepts such as models, tokens, weights, KV cache, quantization and inference. The guide is aimed at readers following the growth of local AI hardware, such as Nvidia's RTX Spark, and is intended to support the publication's ongoing AI coverage.
tomshardware.com
· 2026-10-30
A technical explainer argues that speculative decoding does not make large language model inference faster by cutting the number of floating-point operations performed. In fact, total compute (FLOPs) for a given sequence increases under the technique, since a small draft model proposes tokens that a larger model must then verify in a single parallel forward pass.
jbarrow.ai
· 2026-10-19
LemonSeed Studio is an iPad app that embeds the unmodified upstream Linux amdgpu and amdkfd driver as a PCIDriverKit extension, allowing it to control an AMD GPU connected through a Thunderbolt enclosure. The app also bundles the LemonSeed Engine for on-device LLM inference running on that GPU via its HSA runtime, plus amdgpu_mtopg for live GPU telemetry. The project's source is available via git with recursive submodules, and is dual-licensed under MIT or GPL-2.0-only.
github.com
· 2026-10-11
Salvatore Sanfilippo, the creator of Redis, has released DwarfStar 4 (ds4), a lightweight C-based inference engine for running large open-weight language models locally on high-memory Mac, CUDA and ROCm hardware. The tool supports models including DeepSeek V4, V4.1 Flash, GLM 5.x and Qwen3.8 Flash Next, covering text and vision, with a local API, CLI and native agent, and is released under the MIT license.
dwarfstar.sh
· 2026-10-02
Inception has released Mercury 2.5, an update to its diffusion-based language model that the company says is 40% more intelligent than Mercury 2 while keeping the same low-latency, low-cost performance. The model runs at over 1,100 tokens per second on standard NVIDIA GPUs, supports a 260K-token context window, and launches at an 80% discount, priced at $0.04 per million input tokens and $0.15 per million output tokens.
inceptionlabs.ai
· 2026-09-08
A technical breakdown describes the 'efficient frontier' concept in LLM inference, distinguishing techniques that shift performance along a fixed tradeoff curve—like sacrificing latency for throughput or intelligence for speed—from techniques that expand the frontier itself, creating more overall efficiency. Examples cited include quantization, distillation, pruning, and reasoning-level adjustments, applied in the context of running large agentic coding models such as GLM-5.3 or Kimi K3 with KV cache reuse.
baseten.co
· 2026-09-01
Researchers documented that vLLM's XML-based tool parser for Qwen3 Coder (CVE-2025-9141) fed model-generated arguments directly into Python's eval() function, meaning a language model's own output tokens could execute arbitrary code on the machine running it. A Gemini-based review flagged the change as a critical risk before it was merged, but the lead vLLM maintainer force-merged it anyway.
boydkane.com
· 2026-08-24
GoKawiil