Why routing LLM calls is harder than it looks (lessons from building ai-gateway)

The author details the unexpected complexity of efficiently routing LLM calls, which led to building an AI gateway that decides which model to use per request. This system aims to optimize costs and performance by directing simple prompts to cheaper models and using methods like embedding similarity for routing decisions.