GoKawiilK3 is a frontier quality open model at a fraction of the cost. Even bigger is that it complements Fable predictably, which makes it possible to get the highest quality intelligence by routing tasks.
🧭 tl;dr: We ran Kimi K3 (open) against Fable 5 (closed) on ~1,000 agentic tasks finding:
We achieved 93% accuracy with routing between K3 and Fable. Results were up to ~50X more cost effective than Fable alone on long agentic loops, and consistently lower cost across every use case.
How We Measured
We averaged benchmarks, each aimed at a different kind of work, and ran K3 and Fable 5 through the same harness. About 1,030 tasks in all, in real agent loops.
Family What it tests Tasks SWE Real repo bug-fixes (SWE-bench style) 460 Terminal Long agentic ops: security, crypto, reverse-eng, sysadmin 89 Algorithmic LeetCode / AtCoder-style problems 100 Multi-Language Implementation across six languages 225 Legal A legal-agent benchmark (lawyer-graded tasks) 120
One quick definition before we get into the results. Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling). In a practical router, you don’t get to run your task against multiple models. The router makes a prediction of which model has the best cost and quality trade off, but ultimately it’s a guess.
In this study, oracle routing demonstrated K3 is selected for 72-96% of tasks. This suggests a near-perfect router might be achievable, by learning the difference between day-to-day tasks and the true long tail of frontier work. It will require an order of magnitude more routing data, and real world performance to say definitively.
K3 is a good model.
... continue reading