Let’s talk about model routing

As AI use increases within an organisation, unsurprisingly someone raises the question whether all tasks need to route against the most expensive model. And the right answer, which a lot of organisations are also giving at the moment is no. The most important question here is not if but how you implement a more cost efficient but equally performant routing. Here are three steps to get you started
1️⃣ AI gateway. The first and most important step of this transformation is that you introduce a intermediate layer between the requests from the organisation users and the AI provider of choice. This where you log what users are using the organisational AI for but its also where you will be making the routing decision about which task uses which model. Introducing that layer is easy, you can use an external provider like TrueFoundry or build your own using litellm or similar.

2️⃣ Evals. The second step is constructing a dataset that will allow you to measure performance against real tasks within the organisation. This will allow you to feel confident that the routing decisions you make have small to no impact in performance but a big impact on cost. You can create an eval dataset quite easily nowadays, for example by using a combination of frontier models to given you a first pass of ground truth and then by manually checking a sample of that until you are satisfied with the sample accuracy.

3️⃣ Routing. The third and least important step is routing itself. I say least important because as it turns out you can use quite simply heuristics to get a long way towards a cost efficient routing. You can use a simple easy, medium, hard classification and three model choices like haiku, sonnet, opus. Your heuristic can consist of simple features like length of request, mention of certain keywords like think, presence of multiple questions etc. You can derive the heuristic rules from your actual data with a bit of analysis of the actual requests and a few experiments around heuristic performance. You can eventually graduate to a more sophisticated based routing based on AI but note that this will give you a smaller uplift that you might expect as its an active area still and no definite patterns have emerged.

And just like that you can cut your bills by more than 50% or double usage for the same cost💲