When finance tossed the bill over last month, I thought she'd misread it. Call volume was up 40% month-over-month, but the bill jumped from just over six figures straight to nearly triple. Linearly, we should have paid at most 40% more—so where did the extra come from?
I pulled the last thirty days of call logs and sliced them by model, by endpoint, and by retry count. The problem wasn't "volume"—it was "structure." Four hidden bleeding points, each more insidious than the last.
Let's start with the most glaring one: the flagship model doing grunt work. We had an intent recognition module that only needed a lightweight model, but early on, for convenience, it was hung directly on the most expensive flagship model, and nobody ever touched it. This type of traffic accounted for over 60% of total requests but contributed very little business value. I reran it with model tiering, pushing classification, extraction, and format validation down to small models, leaving only the requests that truly need reasoning to the flagship model—this part of the cost dropped to one-third to one-half of the original. Don't underestimate this move: it doesn't require changing business logic, only routing.
How exactly do you tier? I split it into three tiers by task type: the first tier is classification and intent recognition—short input, short output, simple logic—small models handle it perfectly; the second tier is information extraction and format validation—needs a bit of semantic understanding but no multi-step reasoning—use mid-tier models; the third tier is complex multi-step reasoning, long-form generation, and code completion—leave those to the flagship model. After the switch, I compared: first-tier tasks on small models dropped less than two percentage points in accuracy, but per-call cost fell by nearly 90%. The second tier was more subtle—the F1 gap between mid-tier and flagship models on extraction tasks was under 0.03, yet the cost difference was fivefold. Once you do the math, it's obvious: the extra money we spent before was pure laziness.
The second pitfall is sneakier. Retry storms. Some upstream endpoint had its timeout threshold set too tight, so a little network jitter made the client retry, the retry timed out, and it retried again. In the logs you could see the same request hit four or five times, burning Tokens each time, while the user only saw one failure. This kind of wasted consumption doesn't stand out on the bill, but it can account for around 20%. The fix is simple: change retries to idempotent retries with backoff, loosen the timeout threshold, and add request deduplication on the server side. After that, wasted Tokens were cut by more than half.
Let me expand on a few key parameters here. We loosened the timeout threshold from 3 seconds to 15 seconds, changed retries from unlimited to a maximum of two, and used exponential backoff with jitter—first retry waits 1 second, second waits 2 seconds, with random jitter to avoid thundering herds. On the server side we added a request fingerprint: the same fingerprint is processed only once within 60 seconds, and subsequent requests return the cached result directly. After the change I watched it for a full week: the retry rate dropped from 18% to under 4%, and wasted Token consumption was cut by over 70%. There was an unexpected bonus too: user-side timeout complaints actually went down, because after loosening the threshold, those requests that would have succeeded but were cut off prematurely now return normally.
The third is untrimmed long context. Our chat endpoint stuffed the entire history in by default—by the twentieth turn, input Tokens were more than ten times the first turn. But most of that history is useless for the current answer. With a sliding window plus summary compression, keeping only the most recent few turns of raw text plus a rolling summary, the input side drops 40–60%. One detail here: the summary itself costs Tokens, so you can't compress the window too aggressively—you have to find that inflection point.
We tried several parameter sets and settled on "last five turns of raw text + rolling summary." The rolling summary isn't regenerated every turn but updated every three turns, compressing earlier conversation into a summary under two hundred words. This brought input Tokens down from over eight thousand at the twentieth turn to just over three thousand—a drop of over 60%. But I also tried compressing to only the last two turns, and answer quality clearly degraded—when users asked follow-up questions about context, the model often answered off-topic. So that inflection point really does exist, and you have to test it for your own business scenario. Our rule of thumb: don't let the window go below the last three turns, and don't update the summary more often than every two turns—otherwise the Tokens you save aren't worth the hit to user experience.
The fourth is low cache hit rate. Identical or highly similar requests kept coming in, yet we re-ran inference every time. System prompts, fixed templates, high-frequency Q&A—all perfectly cacheable content—had a hit rate under 10% at the time. After adding a semantic cache layer, duplicate requests return directly; how much this saves depends on your business's repetition rate—for us it was roughly 20–30%.
The semantic cache deserves a couple more words. Exact caching only hits identical requests, but in practice a huge number of requests are "similar in meaning but differently worded." For example, a user asks "how do I get a refund" versus "what's the refund process"—exact matching can't catch it, but semantic caching can. Our approach: vectorize the input, run approximate nearest neighbor search in the cache store, and return the cached result directly if similarity exceeds a threshold. Setting the threshold is critical—too high and the hit rate won't go up; too low and you risk returning irrelevant answers. We settled on 0.92, which pulled the hit rate from under 10% to 34%, with zero user complaints about off-topic answers. System prompts and fixed templates are simpler—just do prefix caching, with a hit rate near 100%.
Stack all four moves together, and when we later reran pay-as-you-go billing plus model tiering, the bill fell back to about 1.2 times pre-growth—while call volume still had that same 40% increase. By the way, along the way we tried token8341 for unified multi-model access—one Key to switch between several model providers, and with routing and billing unified, troubleshooting these structural problems became much easier.
One more easily overlooked point: monitoring and attribution. The reason these four problems went undetected for so long is fundamentally that our bill was only broken down by model and endpoint—not by "task type" and "request fingerprint." Later I added a layer of call tags: every request tagged with task type, whether it was retried, whether it hit cache, and context turn count. When the bill came out again, which link was burning money was obvious at a glance. This tagging system doesn't save money by itself, but without it, you have no idea where the money went.
Pitfall-avoidance reminder: don't just switch to a cheaper model right away. First break down the call structure and look—often the money isn't spent on unit price, but on duplication and redundancy you never noticed. Unit price optimization is the last step, not the first.
Author: Zhao Qiming
Published: November 9, 2026