We expect our APIs to have 99.999% uptime. We'd never tolerate a payment processor that failed 2% of the time. So why do we accept LLMs that routinely break JSON syntax in structured output requests?
Today we're launching Response Healing : a new feature on OpenRouter that automatically fixes malformed JSON responses from LLMs before they reach your application.
Two standout improvements from a week of data:
Gemini 2.0 Flash , our most popular model for structured output with over 1.6 million requests in the past week, saw its defect rate decline by 80% .
, our most popular model for structured output with over 1.6 million requests in the past week, saw its defect rate . Qwen3 235B, one of the most capable open-weight models available, saw its defect rate decline by 99.8%.
The Math That Should Keep You Up at Night
Here's something most developers overlook: if an LLM has a 2% JSON defect rate, and Response Healing drops that to 1%, you haven't just made a 1% improvement. You've cut your defects, bugs, and support tickets in half.
At OpenRouter's scale, we see this compounding effect across billions of tokens daily. A "small" improvement in structured output reliability translates to dramatically fewer 3am pages, fewer angry users, and fewer hours debugging why your agent suddenly stopped working.
This is why we obsess over this problem more than any other gateway. Reliability at the margins is where real production systems succeed or fail.
What We're Fixing
... continue reading