GoKawiil🔌 Engineering
September 7, 2026
Might seem simple on the face of it, but unfortunately it's pain all the way down.
I run Olly, an AI assistant that lives in iMessage, on open source models through OpenRouter. To date Olly's transacted over 18 million messages, roughly a third of those on open models via OpenRouter. That's enough volume to hit every edge case at least once. So here's a list of things I wish I'd known going in.
But first quick vocab: the model is the weights. The provider is who OpenRouter routes you to, they host the model on their GPUs, at their chosen precision, and their "proprietary" optimizations, with their own XML/tool parsers, which means each has a "proprietary" list of bugs too. When you ask for deepseek/deepseek-v4-flash you get one of ~20 companies you've mostly never heard of. They're the same model on paper, but very different models in real life.
Ok, here's a few of the pitfalls you should watch out for.
1. The same model will benchmark very differently
OpenRouter runs per-provider benchmarks on the same model: GPQA Diamond and TAU-Bench Airline (a tool-calling task). Here is today's board for DeepSeek V4 Flash 0731, every provider serving the exact same weights:
DeepSeek V4 Flash 0731, one dot per provider, 2026-09-07 70% 75% 80% 85% 90% 55% 60% 65% 70% 75% 80% GPQA Diamond (knowledge) TAU-Bench Airline (tool calling) DeepSeek: GPQA 90.2%, TAU 81.3% first-party NextBit: GPQA 89.9%, TAU 76.6% Alibaba Cloud Int.: GPQA 89.4%, TAU 76.8% SiliconFlow: GPQA 90.0%, TAU 75.4% NovitaAI: GPQA 89.3%, TAU 76.0% Ionstream: GPQA 87.2%, TAU 78.0% Ionstream GMICloud: GPQA 89.0%, TAU 75.8% Reka AI: GPQA 89.1%, TAU 75.4% Parasail: GPQA 89.0%, TAU 75.4% Baidu Qianfan: GPQA 89.6%, TAU 74.8% Cloudflare: GPQA 88.4%, TAU 75.0% CoreWeave: GPQA 87.1%, TAU 76.0% DeepInfra: GPQA 89.2%, TAU 73.9% StreamLake: GPQA 87.8%, TAU 74.9% Phala: GPQA 88.2%, TAU 74.5% Inceptron: GPQA 87.9%, TAU 74.1% AtlasCloud: GPQA 88.3%, TAU 73.6% Together: GPQA 86.9%, TAU 75.0% Decart: GPQA 87.9%, TAU 73.3% Venice: GPQA 86.8%, TAU 74.2% AkashML: GPQA 88.5%, TAU 72.5% Morph: GPQA 85.8%, TAU 74.9% Wafer: GPQA 84.1%, TAU 76.0% Wafer Ambient: GPQA 86.4%, TAU 73.5% Relace: GPQA 87.2%, TAU 71.7% Io Net: GPQA 84.1%, TAU 74.5% Makora: GPQA 86.9%, TAU 71.7% Mancer: GPQA 85.5%, TAU 70.8% Mancer Sail Research: GPQA 70.8%, TAU 75.2% Sail Research OpenInference: GPQA 70.5%, TAU 70.8% OpenInference Nebius: GPQA 75.6%, TAU 65.3% Nebius DigitalOcean: GPQA 75.3%, TAU 58.4% DigitalOcean OpenRouter's per-provider board for deepseek/deepseek-v4-flash-0731, 2026-09-07. Rolling 32-day average. Hover a dot for the name.
First-party DeepSeek: 90% GPQA, 81% TAU. DigitalOcean, same weights: 75% and 58%. Most hosts cluster 5 to 7 points below first-party on tool calling, and four of them fall off a cliff on knowledge. For an agent TAU is the score that matters and a 20 point swing is not noise. (In July it was worse: Fireworks scored 46% on TAU, a 30 point gap)
... continue reading