Tech News
← Home  ·  All topics

Llm Inference

3 GoKawiil briefs on this topic

Inception launches Mercury 2.5, a faster diffusion language model at cutting price

Inception has released Mercury 2.5, an update to its diffusion-based language model that the company says is 40% more intelligent than Mercury 2 while keeping the same low-latency, low-cost performance. The model runs at over 1,100 tokens per second on standard NVIDIA GPUs, supports a 260K-token context window, and launches at an 80% discount, priced at $0.04 per million input tokens and $0.15 per million output tokens.

Baseten explains how inference engineers trade off latency, throughput and cost for LLMs

A technical breakdown describes the 'efficient frontier' concept in LLM inference, distinguishing techniques that shift performance along a fixed tradeoff curve—like sacrificing latency for throughput or intelligence for speed—from techniques that expand the frontier itself, creating more overall efficiency. Examples cited include quantization, distillation, pruning, and reasoning-level adjustments, applied in the context of running large agentic coding models such as GLM-5.3 or Kimi K3 with KV cache reuse.

vLLM tool-parser bug let AI-generated text trigger code execution

Researchers documented that vLLM's XML-based tool parser for Qwen3 Coder (CVE-2025-9141) fed model-generated arguments directly into Python's eval() function, meaning a language model's own output tokens could execute arbitrary code on the machine running it. A Gemini-based review flagged the change as a critical risk before it was merged, but the lead vLLM maintainer force-merged it anyway.