There is a conversation occurring in enterprise as we reevaluate the era of token maxing. Should our standard AI architecture send every query to a frontier model?
Creating presentation content and analyzing a regulatory contract hit the same endpoint. The bill reflects that, and the trajectory is upward.
The hardware shift is already starting in places. I know one company that recently bought H200s and has Mistral running locally in production at scale for specific workloads. The routing layer that would let that hardware handle a wider range of production traffic in a controlled way is the piece I want to discuss.
I am thinking about a tiered architecture with three model classes behind a router. The router picks the cheapest tier likely to succeed and escalates only when the answer comes back weak.
Routing decisions need observability. Track which queries went to which tier and how often the router escalated. The harder thing to catch is when the cheap tier returns a bad answer the system doesn't flag. Without that data, you are trusting the router instead of measuring it.
That observability is not free. It means an AI gateway sitting between every internal application and the upstream providers, logging each query at the routing layer. You build the gateway before you build the router, because the router has nothing to tune against until the gateway is collecting traffic.
Anyone running this kind of measurement on their AI workloads yet? Curious what you have seen.
Written by Duane Grey
AI Strategy & Implementation
Independent AI consultant helping companies cut through hype and deploy systems that produce real results.