Latest Tech News

Stay updated with the latest in technology, AI, cybersecurity, and more

Filtered by: vllm Clear Filter

Voxtral-Mini-3B-2507 – Open source speech understanding model

Voxtral Mini 1.0 (3B) - 2507 Voxtral Mini is an enhancement of Ministral 3B, incorporating state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation and audio understanding. Learn more about Voxtral in our blog post here. Key Features Voxtral builds upon Ministral-3B with powerful audio understanding capabilities. Dedicated transcription mode : Voxtral can operate in a pure speech transcription mode to maximize p

Mistralai/Voxtral-Mini-3B-2507 · Hugging Face

Voxtral Mini 1.0 (3B) - 2507 Voxtral Mini is an enhancement of Ministral 3B, incorporating state-of-the-art audio input capabilities while retaining best-in-class text performance. It excels at speech transcription, translation and audio understanding. Learn more about Voxtral in our blog post here. Key Features Voxtral builds upon Ministral-3B with powerful audio understanding capabilities. Dedicated transcription mode : Voxtral can operate in a pure speech transcription mode to maximize p

VLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

GitHub | Documentation | Paper LLMs promise to fundamentally change how we use AI across all industries. However, actually serving these models is challenging and can be surprisingly slow even on expensive hardware. Today we are excited to introduce vLLM, an open-source library for fast LLM inference and serving. vLLM utilizes PagedAttention, our new attention algorithm that effectively manages attention keys and values. vLLM equipped with PagedAttention redefines the new state of the art in LL

Life of an inference request (vLLM V1): How LLMs are served efficiently at scale

Life of an inference request (vLLM V1): How LLMs are served efficiently at scale Junhao Li Senior Software Engineer Ubicloud is an open source alternative to AWS. We offer managed cloud services that build on top of PostgreSQL, Kubernetes, vLLM, and others.‍ ‍vLLM is an open-source inference engine that serves large language models. We deploy multiple vLLM instances across GPUs and load open weight models like Llama 4 into them. We then load balance traffic across vLLM instances, run health

Lossless LLM 3x Throughput Increase by LMCache

Redis for LLMs - Infinite and Ultra-Fast LMCache is an LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios. By storing the KV caches of reusable texts across various locations, including (GPU, CPU DRAM, Local Disk), LMCache reuses the KV caches of any reused text (not necessarily prefix) in any serving engine instance. Thus, LMCache saves precious GPU cycles and reduces user response delay. By combining LMCache with vLLM, LMCache achieve