GoKawiilTL;DR
Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per second (RPS) and sub-50 ms p95 time-to-first-audio (TTFA) while maintaining real-time playback on a single NVIDIA H100 SXM.
×
We compare five implementations: ours, vLLM-Omni, SGLang-Omni , VoxServe, and M*, under Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only one to achieve sub-50 ms p95 TTFA. We maintain sub-50 ms p95 TTFA through 10 RPS and keep it below 100 ms even at 20 RPS.
Our system produces approximately 630 characters per second at 10 RPS. At $4.29 per hour for a 1× H100 SXM instance, this translates to ~$2 per 1M characters at full utilization . For comparison, ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M at a higher TTFA.
We open source the implementation and benchmark. Our methodology is explained below.
Interested in custom deployments or want to optimize your multimodal inference workload? Let’s chat
Defining “Real-time” TTS
Let’s start by discussing what a real-time TTS server needs to achieve. We think it’s a four-part problem:
Low Audible TTFA: Time from request dispatch to the first audible sample must be low. Zero underruns: Once playback starts, the client must not run out of buffered audio. Capacity: 1 and 2 must hold as RPS increases. Non-malformed output: Speech must be intelligible.
... continue reading