If you're building the backend for multi-model access, you've almost certainly been through this kind of meltdown: business code stuffed with if (vendor === 'openai') ... else if (vendor === 'claude') ..., a single streaming chat endpoint written with eight hundred lines of adaptation logic, and every new vendor forces you to touch the core path again. The problem isn't you — it's that the APIs of the major LLM vendors were never designed to be unified.
The essence of a model gateway is a layer of protocol translation: collapsing the heterogeneous requests/responses of N vendors into 1 set of internal standard protocol. This article breaks it down in three progressive parts — principles, hands-on practice, and comparison — to explain exactly how the translation layer should be built.
1. Four Categories of Heterogeneity: Why Direct Connections to Multiple Vendors Inevitably Spiral Out of Control
Let's start with the real differences, each with numbers.
Authentication methods: OpenAI uses Authorization: Bearer sk-xxx, a header field of about 51 characters; Claude uses x-api-key; Gemini stuffs the key into the URL's ?key= query parameter; Qwen and ERNIE use Authorization but with different signing algorithms, and ERNIE's access_token has a default validity of 30 days, returning error code 110 when expired. Four vendors, four different approaches.
SSE streaming chunk formats: OpenAI's chunk is data: {"choices":[{"delta":{"content":"x"}}]}, with the end marker data: [DONE]; Claude's event type is event: content_block_delta paired with data: {"delta":{"text":"x"}}, where the field name changes from content to text; Gemini puts the delta in candidates[0].content.parts[0].text, nested 3 levels deep. To parse the same sentence, the business layer has to write four sets of parsers.
Error code systems: This is the most painful part. OpenAI uses HTTP status codes plus error.type, with rate limiting as 429; Claude returns 429 for rate limiting but the type in the body is rate_limit_error; Qwen's rate limiting may return a 200 status code but carry code: "Throttling" in the body; ERNIE uses error_code: 17 to indicate quota exceeded. The same event, expressed four ways.
Token counting conventions: OpenAI's usage.total_tokens = prompt + completion, with both Chinese and English split by BPE; Claude returns input_tokens and output_tokens separately, with finer-grained character splitting for Chinese — for the same 500-character Chinese passage, the token counts from the two vendors can differ by 20% to 35%. For usage-based billing, this difference translates directly into billing errors.
Connecting directly to four vendors inflates the business-layer code to 3 to 4 times that of a single vendor, and maintenance cost rises linearly with the number of vendors.
2. Translation Layer Design: Unified Format, Stream Normalization, Error Code Mapping, Usage Aggregation
The translation layer must do four things, and none can be omitted.
Unified request/response format: Define an internal standard Schema, with fields aligned to the OpenAI style (because its ecosystem is the broadest). Incoming external requests are first converted into the internal format, then the adapter converts them into each vendor's format. The adapter pattern — one class per vendor, implementing the two methods buildRequest() and parseResponse().
Streaming chunk normalization: The core is converting each vendor's SSE chunk into a unified {delta: string, finish_reason: string|null}. The approach is to write a state machine that dispatches to different parsers by vendor type and outputs a unified structure. The key point is boundary handling: SSE's \n\n separators, the data: prefix, and multi-line data concatenation all need to be swallowed inside the parser. A pitfall I've hit is Claude's ping event and message_stop event — if you don't explicitly ignore them, empty deltas get pushed to the frontend and cause UI flickering.
Error code mapping table: Maintain a vendorCode -> unifiedCode mapping, unified into 6 categories: AUTH_FAILED, RATE_LIMITED, QUOTA_EXCEEDED, INVALID_REQUEST, UPSTREAM_ERROR, TIMEOUT. Whatever the upstream returns, the translation layer looks it up and converts it into a unified code, and the business layer only recognizes these 6.
Usage aggregation: After each call, convert each vendor's token convention into internal standard tokens, and persist it with vendor, model, and request_id for later reconciliation and cost analysis. Note the convention differences here — Claude's output_tokens and OpenAI's completion_tokens cannot be added directly; they must be normalized to the same baseline using conversion coefficients.
In terms of layered architecture, I usually split it into four layers: access layer (auth + rate limiting), routing layer (model selection), translation layer (protocol conversion), and adapter layer (vendor SDK calls). The translation layer is pure functions, no IO, making it easy to unit test.
3. Hands-On Code: Change One Line of base_url to Access an Aggregation Gateway
If you don't want to write the translation layer from scratch, using an existing AI API aggregation platform is the more pragmatic choice. In our project we've used token8341 — it's compatible with the OpenAI SDK, you switch by changing one line of base_url, and it does the translation layer's work for you.
from openai import OpenAI
client = OpenAI(
api_key="your-key",
base_url="https://api.token8341.com/v1" # only change this line
)
resp = client.chat.completions.create(
model="deepseek-v3", # can also switch to gpt-4o / qwen-max / claude-4-sonnet
messages=[{"role": "user", "content": "Explain SSE"}],
stream=True
)
for chunk in resp:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="")In this code, SSE chunk normalization, error code mapping, and usage aggregation are all done on the gateway side, and what the business layer receives is the standard OpenAI format. Multi-model routing is also handled — the gateway can automatically select the optimal model by task, e.g., route Chinese long-form text to Qwen, reasoning tasks to DeepSeek, and cost-sensitive scenarios to Doubao.
Pitfall reminder: after switching base_url, don't pass through vendor-specific parameters directly. For example, OpenAI's logit_bias and Claude's top_k are not necessarily fully supported by the gateway; passing them through may result in them being silently ignored, so you think they're in effect when they aren't. Before going live, run a stress-test script to check parameter compatibility.
4. Comparison: Multi-Model Routing vs Single-Model Binding
Here's a set of numbers from our own testing. Single-model binding (direct to OpenAI): integration takes about 2 person-days, failover requires code changes and redeployment, average recovery time is 15 minutes, and there's no room for cost optimization. Multi-model routing (via gateway): integration takes about 0.5 person-days, failover happens automatically, recovery time is under 3 seconds, and costs can be reduced by about 30% to 40% by routing tasks appropriately.
In terms of maintenance complexity, the adaptation code for connecting directly to 4 vendors is about 2000 lines; after going through the gateway, the business-layer adaptation code drops to under 200 lines, a 90% reduction. On latency, the gateway adds one hop, increasing P99 by about 20 to 40ms — an acceptable overhead compared with the maintenance cost and switching flexibility it saves.
Of course, a gateway is not a silver bullet. If your business uses only one model and will never switch, direct connection is simpler. But as long as multiple models, multiple vendors, and cost optimization are involved, the translation layer is essential infrastructure. Model gateway, LLM API, and AI API aggregation — these three things are essentially three ways of describing the same layer of problems.
One final reminder: the unit test coverage of the translation layer must be maxed out, especially the boundary cases of streaming chunks — this is a high-incidence area for production incidents.
Author: Liu Zhiyuan
Published: October 28, 2026