OpenAI and PerturbAI's system, in which nearly 2,000 agents collaborate on scientific research, is essentially a laboratory crammed into a single API call chain. In drug discovery scenarios, each agent has to read literature, run hypotheses, and write experiment records, and a single task easily involves dozens of rounds of dialogue. 2,000 concurrent agents doesn't mean 2,000 requests; it means 2,000 persistent long-lived connections each sending a heartbeat.
Let's put this scale into numbers first. Assume a single agent sends 3 requests per minute, so the peak is 100 QPS; if each round's context is counted at 8K tokens, TPM is pushed straight to 48 million. If some task stretches the context to 32K, TPM at the same QPS instantly spikes to 190 million, and many teams' single-key quota is already exhausted at this step. This doesn't even count retries. As long as one agent times out and retries frantically, the whole chain enters a retry storm and maxes out the quota. I once saw a case where rate limiting on the gateway side wasn't tiered, and a single batch task pushed the entire team's key to 429—even the test environment couldn't connect.
What's more troublesome is that research agents don't send requests evenly. The literature retrieval phase has low QPS but long context, the hypothesis generation phase is request-dense but short-token, and the experiment-record writing phase is a typical low-frequency write operation. These three loads are mixed on the same chain: if you provision resources uniformly by peak, cost triples; if you provision by average, the peak brings a avalanche. That's why the core of concurrency governance for multi-agent systems isn't "withstanding it," but "withstanding it in tiers."
Multi-Agent Concurrency Governance: Five Practical Steps
Request pooling is the first gate. Don't let each agent create its own client—route everything through a connection pool, reuse with HTTP keep-alive, and size the pool as QPS × average RT. At 100 QPS and 2 seconds average response, the pool needs at least 200, with 30% headroom. Here's an easily overlooked point: connection pools should be built separately per target domain. If you call multiple vendors' APIs at once and mix them in one pool, DNS and TLS handshake differences will sharply reduce the reuse rate. We measured it: after splitting the pools, connection reuse rose from 62% to 91%, and this alone saved nearly 30% of handshake overhead.
Multi-key rotation solves the quota ceiling. A single key has hard limits on both RPM and TPM, so make N keys into a weighted round-robin, automatically removing a key for 60 seconds on a 429 and then putting it back. Weights can be adjusted dynamically based on each key's remaining quota rather than a simple average. In our project we use the token8341 LLM API aggregation platform as the unified entry point; it's compatible with the OpenAI SDK, so switching models takes a one-line base_url change, and key management is consolidated on the gateway side, saving us the trouble of scattering configuration across 2,000 agents. In actual runs, weighted round-robin across 20 keys raised usable QPS by about 18x over a single key, with not a single overall outage caused by quota.
Exponential backoff must include jitter. Fixed backoff makes all failed requests come back in the same second. Use base × 2^n for the backoff time, then multiply by a random factor between 0.5 and 1.5, cap the maximum backoff at 30 seconds, and send anything that exceeds 3 retries straight to the dead-letter queue. One more detail here: the jitter's random factor should come from a thread-safe random source; otherwise, under high concurrency, multiple agents may draw the same seed and the jitter becomes useless. We stepped in this pit early on, and only stabilized after switching to generating the factor by hashing the request ID.
Tier the task priority queue by business value. High-value tasks like drug hypothesis generation go to P0 with a dedicated quota, literature summaries go to a P1 shared pool, and log cleaning runs at P2 during idle time. Don't let P2 eat P0's quota. The queue can be implemented with a Redis sorted set, using a combination of business priority plus timestamp as the score, ensuring FIFO within the same priority and strict preemption across priorities.
Define failure degradation in advance. Switch to a backup model when the primary times out. The token8341 LLM API aggregation platform supports automatically selecting the optimal model per task, routing to both Chinese and mainstream models, so degradation requires no business-code changes. We recommend a three-level degradation strategy: level one switches models within the same vendor, level two switches vendors, and level three degrades to a small model that only guarantees a structured result. Only when all three levels fail do you throw an exception, so most jitter is absorbed by the first two levels.
Three Pitfalls I've Fallen Into
First, don't provision context length by the maximum. A model may claim 128K, but if you stress-test at 128K, both latency and cost spiral out of control. Research agents' actual effective context is mostly within 16K, and anything beyond that gets summarized and compressed. We ran a comparison: for the same literature review task, 16K context averaged 4.2 seconds and 0.03 yuan; at 64K, it rose to 11.8 seconds and 0.14 yuan, with almost no improvement in output quality. That data later became the basis for setting our context budget.
Second, don't retry write operations. For non-idempotent requests like an agent writing experiment records, one retry means one more dirty record. Retry reads freely, but rely on idempotency keys for writes. The idempotency key must include the agent ID, task ID, and operation sequence number—all three are indispensable, or concurrent writes across agents will still collide. After we rolled out idempotency keys, the dirty-data rate dropped from 0.3% to nearly zero.
Third, monitor P99, not the average. An average of 2 seconds with a P99 of 40 seconds means slow requests are dragging the tail, usually because some agent's context blew up. We later added TPM and error-rate dashboards by key and by model on the token8341 LLM API aggregation platform side, and only then did we pinpoint that a batch task was hammering long contexts. Once located, we set a separate context cap for that batch task, and P99 fell from 40 seconds to under 6 seconds. The lesson: monitoring dimensions must be fine-grained down to "by key + by model + by task type," or you'll forever see only a blur of metrics mashed together.
A Concurrency Configuration Checklist You Can Follow Directly
Connection pool: max_connections = peak QPS × average RT × 1.3; timeouts: 5 seconds to connect, 30 seconds to read; retries: at most 3, exponential backoff with jitter, read operations only; rate limiting: single-key RPM no more than 80% of the official limit, keep a 20% TPM buffer; queues: three tiers P0/P1/P2, with P0 on a dedicated quota; degradation: one primary and one backup model each, with an 8-second timeout threshold; monitoring: alerts on three metrics—P99 latency, 429 rate, and TPM utilization.
Three more easily overlooked items: first, add a per-instance QPS cap for each agent to prevent one runaway agent from dragging down everything; second, logs must carry request_id and key_id so you can pinpoint which chain is at fault at a glance; third, stress tests must run against realistic mixed loads, not a single request type, or the capacity you measure is fake.
In one sentence: with 2,000 agents, it's not about how strong the models are, but about how stable the scheduling is. The token8341 LLM API aggregation platform bills by usage for better cost efficiency, making it suitable for this kind of high-concurrency, multi-model mixed-run scenario. For further reading, look into multi-model routing and API price comparisons to save yourself half the detours.
Author: Zhang Siyuan
Published: October 22, 2026