A new open-source project called Flide was shown on Hacker News, offering fast inference infrastructure for Jev-style decision models via an exact state-forking technique. Its creators say it runs up to 9.5 times faster than existing engines vLLM and SGLang while preserving model-only answers, and version 0.1.10 adds support for several new decision models including Ainvest-Decision, Darwin-27B-RSI-Decision and KnowLine-31B-Gen1.5, the latter now able to read images.
github.com
· 2026-11-09
vLLM, an open-source inference engine, has shipped a server-side flag in version 0.31 that enables constrained decoding, a technique forcing model output to match a required schema. The post's authors say that serving Kimi K2.6 on vLLM 0.28 produced correctly formatted tool calls only 92% of the time, but enabling the new flag raised accuracy to 100% for newer Kimi and DeepSeek models.
ubicloud.com
· 2026-10-26
BottleCap AI has published a fifth ThinkingCap model, this time trained on Google's Gemma-4-26B-A4B-it, cutting the number of 'thinking' tokens the model uses by up to 69.1% and an average of 38.9% across twelve benchmarks. The company says accuracy drops by only 0.15 percentage points, and that looping behavior fell from 3.6% of responses to 0.8%. A PRO version is also offered, claiming up to 87.6% fewer thinking tokens (62% average) for enterprise customers.
bottlecapai.com
· 2026-10-23
Google and Meta jointly released a PyTorch-native version of the SGLang inference engine, called TorchTPU, letting Google's tensor processing units serve AI workloads without requiring a switch to Google's JAX framework. Meta's PyTorch-TPU lead Zongwei Zhou said the engine is already running in production at Meta, built with help from RadixArk, the startup founded by SGLang's creators. A matching PyTorch-native version of the vLLM inference engine was also released as open source.
tomshardware.com
· 2026-10-22
AMD published results from experiments running speculative decoding in vLLM on its MI300X and MI355X Instinct GPUs using the ROCm software stack. The team compared five drafting approaches—native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark—which differ in how draft tokens are generated and how they receive signals from the target model.
vllm.ai
· 2026-09-07
Researchers documented that vLLM's XML-based tool parser for Qwen3 Coder (CVE-2025-9141) fed model-generated arguments directly into Python's eval() function, meaning a language model's own output tokens could execute arbitrary code on the machine running it. A Gemini-based review flagged the change as a critical risk before it was merged, but the lead vLLM maintainer force-merged it anyway.
boydkane.com
· 2026-08-24
GoKawiil