GoKawiilOlly's builder details how OpenRouter provider swaps cause hidden model quality drift
The developer behind the iMessage AI assistant Olly, which has processed over 18 million messages with roughly a third routed through OpenRouter's open-source models, published a technical breakdown of problems encountered at scale. The core issue: OpenRouter can send identical model requests to any of about 20 different hosting providers, each running the same weights but with different precision, optimizations and tool parsers, producing measurably different benchmark results. For DeepSeek V4 Flash, GPQA Diamond and TAU-Bench Airline scores varied by several points across providers on the same day.