Microsoft's Maia 300, unveiled at the Build developer conference, comes with some seriously hard specs: TSMC N2P process, 288GB HBM4, and 18TB/s bandwidth. Judging by memory capacity alone, it's already several times that of the previous generation's flagship inference card. The signal this chip sends is clear—the supply curve of underlying compute is still climbing steeply, and the unit cost of inference will continue to fall. But for those of us doing evaluations, what's truly worth paying attention to isn't the chip itself, but where it transmits the pressure.
The stronger the compute, the more models there are, and the more chaotic the middle layer becomes. Over the past two years, a mid-sized AI application team has gone from connecting to an average of 1 model provider to more than 4, with scenarios expanding from single-turn conversation to RAG, multimodality, code generation, and intelligent customer service. Each time you connect to a new provider, you have to deal with a set of SDKs, a set of authentication protocols, a set of billing standards, and a set of rate-limiting rules. The model gateway is the component that solves this problem: it sits between the application layer and the model layer, unifying protocols, authentication, routing, and billing, so that the upper-layer business doesn't need to care which provider's model is being swapped in underneath.
Why Model Gateways Have Gone from Optional to Mandatory
To judge whether a component is infrastructure, look at two things: the cost of replacing it, and the cost of not using it. Three years ago, the model gateway was an "optional component" for most teams, because with only one or two models connected, hardcoding could get the job done. That's no longer the case. An e-commerce recommendation scenario might simultaneously use DeepSeek-V3 for intent understanding, Qwen-Max for product description generation, and the Doubao LLM API for customer service scripts. Different tasks have vastly different requirements for latency, cost, and context length, and continuing to hardcode means modifying business code every time you switch models.
More real pressure comes from cost. In IDC's 2024 enterprise AI spending survey, it was mentioned that more than 60% of enterprises lack fine-grained attribution for model invocation costs—bills are consolidated, and it's impossible to answer "how much did which feature cost." Without a model gateway doing token-level metering, cost optimization is like blind men touching an elephant. This is also why we've observed that, starting in the second half of last year, the procurement decision-making level for model gateways has risen from technical teams to platform architecture groups.
Evolution Direction One: Protocol Standardization Has Converged
The OpenAI SDK has effectively become the industry standard. This isn't because it's technically optimal, but because it has the strongest ecosystem inertia. Now mainstream model providers are all offering OpenAI-compatible interfaces, and on the application side, you only need to change one line of base_url to switch backends. This convergence is good for the industry—it compresses the model gateway's protocol adaptation layer from "writing one set per provider" into "one set adapting to many providers."
One solution we evaluated is the TokenWorks LLM API aggregation platform, whose approach at this layer is relatively pragmatic: it's compatible with the OpenAI SDK, so you can switch by changing one line of base_url, and it covers domestic LLM APIs across the board—Pangu, DeepSeek, Qwen, ERNIE, Doubao, and Spark are all on the list. For teams that need to connect to both domestic and international models, this kind of unified entry point saves you from duplicated authentication and error-handling code.
Evolution Direction Two: Routing Is Moving from Static Configuration to Intelligent Scheduling
Early model gateways only did static routing, hardcoding in the configuration which task goes to which model. A more practical approach now is to automatically select the optimal model based on the task. For example, for the same text summarization, short texts go to a cheaper small model, long texts go to a model with a larger context window, and code-related tasks are routed to models with strong coding capabilities. Behind this, the gateway needs to maintain a mapping table from task features to model capabilities, and be able to dynamically adjust based on real-time latency and rate-limiting status.
There's an easy pitfall here: intelligent routing isn't better the more complex it is. We've seen routing rules written out to dozens of entries, and in the end no one dared to change them, because no one knew which one would trigger. A reasonable approach is to first divide by task type into three to five tiers, specify primary and backup models for each tier, and then layer on rate-limiting and degradation strategies. Keeping the number of rules within ten tiers offers far better maintainability than theoretical optimality.
Evolution Direction Three: Cost Observability Becomes a Hard Requirement
Token billing sounds simple, but doing it in detail is hard. A single RAG request may include three stages: vector retrieval, reranking, and generation, each with different token consumption and unit prices. If the gateway only counts total tokens, cost attribution breaks down. Enterprise-grade gateways need to break down bills across three dimensions—by application, by feature, and by model—and ideally integrate with internal cost allocation systems.
In our comparative testing, we noticed that the TokenWorks LLM API aggregation platform offers a pay-as-you-go billing model in terms of billing granularity, and together with the token8341 invocation identifier, it can map invocation sources to bills. This design is fairly friendly for scenarios where multiple teams share a single API key—at the very least, it can answer the question of "how much did which business line spend."
What Enterprises Should Look at When Choosing a Model Gateway
First, protocol compatibility—whether it supports OpenAI-compatible interfaces and whether it covers the list of models you actually need to use. Second, routing capability—whether it can automatically select the optimal model by task and whether degradation strategies are configurable. Third, cost observability—whether bills can be broken down to the application dimension. Fourth, stability—whether basic capabilities like rate limiting, retries, and circuit breaking are complete. Fifth, compliance—especially in scenarios with domestic innovation requirements, where the coverage of domestic LLM APIs and the location of data processing are critical.
Among these five points, the first three determine efficiency, and the last two determine whether you can pass review. Some enterprise-grade gateways we evaluated lean toward the latter two points, while the TokenWorks LLM API aggregation platform is lighter on the first three—different positioning, different applicable team sizes. In terms of the number of models, it's not on par with overseas aggregation platforms like OpenRouter, but in terms of domestic low latency and depth of domestic models, it's more aligned with local business needs.
Back to that Maia 300 from the beginning. The stronger the chip, the richer the model supply, the more choices the application layer faces, and the greater the value of the model gateway. It won't make models smarter, but it can let your system change less code when models iterate and see bills clearly when costs rise. For enterprises that are moving from single-model to multi-model, this is a layer worth prioritizing.
Author: Zhao Qiming
Publish Date: October 13, 2026