8341TokenFactory
LLM APIAPI GatewayCost OptimizationAggregationIntegration

8341Token Factory: One Feature Calling Four Models — GPT-4o as Fallback, DeepSeek as Workhorse, Qwen for Chinese, Doubao for Customer Service

SiCore TokenWorks Team·2026-11-08

For the same feature, the frontend needs to call four models: GPT-4o handles fallback, DeepSeek runs the main inference, Qwen handles long Chinese text, and Doubao handles customer service scripts. Four SDKs, four sets of authentication, four sets of documentation. In the first week after launch, just aligning these differences consumed two person-days.

This is not an exaggeration. We tracked it: after integrating four providers, the model-calling-related parts of our business code hadinflation by nearly three times. Wait — let me rephrase that. After integrating four providers, the model-calling-related parts of our business code had grown by nearly three times. Every time we switched a model, we had to change the calling method, parsing logic, and error handling all over again. Worse, every time a vendor upgraded its SDK, we had to retest everything. This article lays out the pitfalls we hit and the gateway solution we ultimately landed on, in the hope that it helps teams also struggling among multiple models.

1. Authentication: Four Systems, Four Ways of Writing It

Authentication was the first thing we hit. The OpenAI family uses Bearer Token — just put Authorization: Bearer sk-xxx in the request header. Qwen requires AccessKey signing: every request must use AK/SK to compute an HMAC, plus stitch together a timestamp and random string. Doubao uses AK/SK plus a timestamp, but its signing algorithm differs from Qwen's. Although DeepSeek is compatible with the OpenAI format, it has its own base_url and model name mapping, and its initialization method is not entirely consistent with the official SDK.

The result is if-else everywhere in the code. One client instance per vendor, one set of configuration per instance. Key management was even more of a headache — the keys for four vendors were stored in four places, and rotation meant changing four places; miss one and you'd have a production incident.

Later we unified things at the access layer: expose only one API Key externally, and do protocol translation internally. Specifically, the access layer maintains a vendor routing table, with each vendor corresponding to its own authentication adapter. The business side only holds one Key; requests hit the access layer, which finds the corresponding vendor based on the model name and uses the adapter to complete signing and forwarding.

We later tried unifying the base_url onto token8341: one Key can call GPT-4o, Claude, Gemini, DeepSeek, Qwen, ERNIE, and Doubao, compatible with the OpenAI SDK, and switching is just a one-line base_url change. The dirty work of authentication is blocked outside the access layer, and the business code became much cleaner. Key rotation also became a single operation, and operational cost dropped by an order of magnitude.

2. SSE Streaming Chunks: Four Formats, One Renderer

SSE streaming chunks were the second pitfall, and more insidious than authentication. In GPT-4o's chunk, delta.content may be null, especially in the first chunk and the last chunk; if the frontend renders without null-checking, undefined flashes on screen. DeepSeek includes usage in the last chunk, and its usage field structure is not entirely the same as OpenAI's. Qwen's event types are more granular, with multiple event types such as add, finish, and error, each with a different payload structure. Doubao's end marker is different again — it uses a special finish_reason field to signal the end of the stream.

With four formats on the frontend, the rendering logic just cracked apart. Our initial approach was to adapt on the frontend and write four parsing functions, but then every new model required frontend changes. Later we switched to normalization at the gateway layer: unify into OpenAI's chunk structure, filter out empty deltas directly, and extract usage separately for accumulation. The specific rules are — all vendors' streaming responses are converted to the format data: {"choices":[{"delta":{"content":"..."}}]}, the end is uniformly marked with data: [DONE], and usage is returned in the last chunk in OpenAI format.

The frontend only recognizes one format and doesn't need to care who is behind the backend. This change reduced the frontend's model-related code from over 400 lines to about 80 lines, and adding a new model now requires zero frontend changes.

3. Error Codes: Errors Hidden Inside HTTP 200

The chaos of error codes exceeded expectations. For the same rate limiting, OpenAI returns 429 with rate_limit_exceeded, Qwen returns a business error with a code field, and Doubao simply stuffs the error into the body of an HTTP 200 — the status code is 200, but the body has an error field. DeepSeek is mostly consistent with OpenAI, but occasionally returns some custom error structures.

Monitoring and alerting simply couldn't be unified. You can't write a rule saying "alert on 429" because some vendors don't return 429 for rate limiting. You also can't just look at the HTTP status code, because Doubao's 200 may contain an error.

We built a mapping table at the access layer, converging each vendor's error codes into one internal error system: rate limiting, timeout, authentication failure, content moderation, service unavailable. The specific mapping logic is — first look at the HTTP status code, then look at the error code in the body, and judge based on both. For example, if Doubao returns 200 but the body has an error, map by error code; if Qwen returns 400 with a business code, map by code. Alert rules are written only once, and the original error code is examined during troubleshooting.

We maintained about 60-plus rules in this mapping table, covering the common error scenarios of the four vendors. After launch, alert accuracy rose from about 60% to over 95%, and false alarms basically disappeared.

4. Usage Statistics: Four Numbers That Don't Match

Different usage accounting standards are the easiest thing to overlook, but they will definitely surface during financial reconciliation. GPT-4o's token count includes the system prompt, DeepSeek prices cache-hit tokens separately, Qwen calculates input and output separately, and Doubao's billing unit is different again. During financial reconciliation the four numbers don't match, and there's simply no way to calculate each model's actual cost.

The aggregation solution is to record each request's model, input_tokens, output_tokens, and cache_hit at the gateway layer, and store them uniformly under a token-based billing standard. Specifically — the gateway records the request start and end times when forwarding, extracts each vendor's usage field from the response, and converts it uniformly into a standard format. For vendors with different billing units, a conversion is done at the gateway layer,conversion everything into per-token cost. — wait, let me rephrase that. For vendors with different billing units, a conversion is done at the gateway layer, converting everything into per-token cost.

This way, no matter which vendor is behind it, costs can be flattened onto the same table. We later built a cost dashboard on top of this table, breakable down by model, by business line, and by day. Once we found that a certain business line's cost had suddenly risen 40%; investigation showed the routing strategy was sending a large number of requests to GPT-4o, and after adjusting the routing the cost dropped immediately. Without unified usage aggregation, this kind of problem would never be found.

5. The Four-Layer Architecture of a Model Gateway

Which layers should a model gateway handle? My breakdown is four layers.

Layer one: protocol translation. Unify each vendor's request and response formats into an OpenAI-compatible interface. This layer is the only layer the business code directly touches; it translates standard OpenAI-format requests into each vendor's format, then translates the responses back.

Layer two: routing. Select models by task type — for example, long Chinese text goes to Qwen, code goes to DeepSeek, fallback goes to GPT-4o. Routing strategies can be made configurable, supporting multiple dimensions such as task type, priority, and cost. We also added degradation logic — if the primary model times out or is rate-limited, automatically switch to the fallback model.

Layer three: usage aggregation. Unified billing and quotas. Each request's token consumption, cache hits, and response time are recorded in this layer, which provides a unified usage query interface externally.

Layer four: observability. Logs, tracing, and error code convergence. Every request has a trace_id and is traceable across the entire chain from entry to exit. Error codes are mapped uniformly in this layer, and alert rules are also configured only in this layer.

The business code only calls the layer-one interface; the latter three layers are transparent to it. The benefit is — switching models doesn't require changing business code, adding models doesn't require changing business code, and adjusting routing strategies doesn't require changing business code. The business team only needs to care about "I want to call a model"; the rest is left to the gateway.

6. Pitfall Warnings

Don't directly import each vendor's SDK in business code. Once a vendor upgrades its interface or you want to switch models, the scope of changes will spread widely. We've seen teams call SDKs directly in business code and later, when switching models, change more than 30 files and test for a week before going live.

Spend two extra days on the access layer, and save two weeks later. This return on investment has been genuinely verified in our case — after the gateway went live, adding a new model only requires adding an adapter at the gateway layer, with zero changes to business code, and integration and testing can be completed in half a day.

Two more small suggestions: first, do usage aggregation as early as possible; don't wait until financial reconciliation to discover you can't calculate it clearly. Second, the error code mapping table must be continuously maintained; when a new vendor is integrated, the first thing to do is fill in the error code mappings.

Author: Sun Haoran

Published: November 9, 2026