GoKawiilWhen LLM responses are too slow for your realtime use case, you may be tempted to pay double the cost for a faster service tier. Anthropic’s Priority tier, OpenAI’s priority processing, Gemini’s priority inference, whatever your LLM provider calls it. There’s a simpler solution: send every request twice and take the faster response.
Why tail latency matters for voice agents
Our voice agent at HOAi answers phone calls. Every turn in a conversation makes an LLM request. Most responses come back within 1.5 seconds, but occasionally one takes 10 to 20 seconds. On a phone call, that’s 10 seconds of awkward silence, and after enough silence, the caller hangs up on our agent.
This happens more often than you’d think. A typical phone call has 20 to 30 turns. If 1% of LLM requests are catastrophically slow, a 25-turn call has roughly a 22% chance of hitting a long silence.
Priority tier vs. sending each request twice
We had two options.
Upgrade to OpenAI’s priority tier and pay 2x cost per token for faster, more consistent responses. Stay on standard tier, but send every request twice and take the faster response.
We replayed 50 real production requests against both setups and tracked two metrics: time to first token (when the agent starts speaking) and time to complete response (when it can act on tool calls).
Time to first token:
Priority tier Standard tier, sent twice median 0.61s 0.58s p95 1.04s 0.68s p99 4.2s 1.2s
... continue reading