GoKawiilWe don’t believe in model routing anymore. For most use cases, sticking to a single battle-tested model is the best thing you can do.
Recently, there’s been huge hype around AI model routers that select the model that will respond to your request on the fly. There have been many launches in recent weeks with similar promises of reducing inference costs. We had our LLM router too, and decided to remove it.
Some context first: we launched the Manifest LLM router in March as a key feature in our LLM gateway, and we deprecated it in June, shutting it down for good on September 1st. Our router was classifying each request into one of four different tiers of complexity: simple, standard, complex and reasoning.
Like most LLM routers, ours was made for cost reduction. Why call a powerful, and therefore expensive, model for a simple task? Routing to the most cost-effective model seems like a natural solution, right? Not that simple. After four months of usage across 7000 cloud users, we saw mixed results and a lot of GitHub issues and discussions about it. Let’s dive into the main problems.
Complexity cannot be deduced from the prompt alone
The prompt alone does not contain the whole task; it is just the trigger. A lot of the context that determines complexity is only discovered later through tool calls, web searches, and so on.
Let’s take an example: “evaluate the tests for the repo $GIT_REPO and improve them” can be a very simple task if you mention a personal website written in plain HTML5; or an incredibly complex task if you target the Linux kernel repo.
Cache is more effective than routing for reducing costs
Cache reads are between 75% and 90% cheaper than uncached inputs. System prompts and conversation history often represent a lot of tokens. Prefix cache works extremely well for those because they sit at the beginning of the prompt.
A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it. In other words, the router will do its job by, ironically, not doing it.
... continue reading