The system from OpenAI and PerturbAI, where nearly 2000 agents collaborate on scientific research, is essentially putting an entire laboratory inside a single API call chain. In the drug discovery scenario, each agent has to read literature, run hypotheses, and write experiment logs — a single task can easily involve dozens of rounds of conversation. 2000 concurrent agents doesn't mean 2000 requests; it means 2000 long-lived connections with a continuous heartbeat.
Let's put that scale into concrete numbers first. Assume a single agent sends 3 requests per minute — at peak that's 100 QPS; with each round of context at 8K tokens, TPM is pushed straight to 48 million. If some task stretches the context to 32K, at the same QPS the TPM instantly spikes to 190 million, and many teams' single-key quota is already exhausted at this point. And that's before counting retries. All it takes is one agent that goes wild retrying after a timeout, and the whole chain enters a retry storm that maxes out the quota. I've seen a case where the gateway-side rate limiting wasn't layered, and a single batch job drove the entire team's keys to 429 — even the test environment couldn't connect.
What's worse, research agents don't send requests evenly. The literature retrieval phase has low QPS but long context; the hypothesis generation phase is request-dense but with short tokens; the experiment log writing phase is a classic low-frequency write operation. Three workloads mixed on the same chain: if you provision resources uniformly by peak, costs triple; if you provision by average, the peak brings a avalanche. That's why the core of concurrency governance for multi-agent systems isn't "withstanding it" — it's "withstanding it in layers."
Multi-agent concurrency governance: five practical steps
Request pooling is the first gate. Don't let each agent create its own client — route everything through a connection pool, reuse HTTP keep-alive, and size the pool by QPS × average RT. At 100 QPS and 2 seconds average response, the pool needs at least 200, with 30% headroom. Here's an easily overlooked point: connection pools should be built separately per target domain. If you're calling multiple vendors' APIs at the same time, mixing them in one pool causes the reuse rate to plummet due to differences in DNS and TLS handshakes. We measured this: after splitting the pools, the connection reuse rate went from 62% to 91% — that alone saved nearly 30% of handshake overhead.
Multi-key rotation solves the quota ceiling. A single key has hard limits on both RPM and TPM. Turn N keys into weighted round-robin, and on a 429 automatically remove a key for 60 seconds before putting it back. Weights can be adjusted dynamically based on each key's remaining quota rather than a simple average. In our project we use the SiCore TokenWorks LLM API aggregation platform as the unified entry point — it's compatible with the OpenAI SDK, so switching models is a one-line base_url change, and key management is centralized at the gateway, saving us the trouble of scattered configuration across 2000 agents. In actual runs, weighted round-robin across 20 keys improved usable QPS by about 18x over a single key, with not a single full outage caused by quota.
Exponential backoff must include jitter. Fixed backoff makes all failed requests come back in the same second. Use base × 2^n for the backoff time, multiplied by a random factor between 0.5 and 1.5, cap the maximum backoff at 30 seconds, and send anything that fails more than 3 retries straight to the dead-letter queue. One more detail here: the jitter's random factor should come from a thread-safe random source — otherwise, under high concurrency, multiple agents may draw the same seed and the jitter stops working. We hit this pitfall early on and only stabilized after switching to generating the factor by hashing the request ID.
Task priority queues layered by business value. High-value tasks like drug hypothesis generation get a dedicated P0 quota; literature summaries go to a shared P1 pool; log cleaning runs on P2 during idle time. Don't let P2 eat P0's quota. The queue can be implemented with a Redis sorted set, with the score combining business priority and timestamp to guarantee FIFO within the same priority and strict preemption across priorities.
Failure degradation must be decided in advance. When the primary model times out, switch to the backup. The SiCore TokenWorks LLM API aggregation platform supports automatically selecting the optimal model per task and can route to both domestic and mainstream models, so degradation requires no business code changes. I recommend a three-tier degradation strategy: tier one switches models within the same vendor, tier two switches vendors, tier three degrades to a small model that only guarantees a structured result. Only when all three tiers fail do you throw an exception — that way most jitter gets absorbed by the first two tiers.
Three pitfalls I've hit
First, don't provision context length at the maximum. A model may claim 128K, but if you load-test at 128K, both latency and cost spiral out of control. Research agents' effective context is mostly within 16K; anything beyond that gets summarized and compressed. We ran a comparison: for the same literature review task, a 16K context averaged 4.2 seconds and 0.03 yuan; pushed to 64K, it rose to 11.8 seconds and 0.14 yuan, with almost no improvement in output quality. That data later became the basis for setting our context budget.
Second, don't retry write operations. Non-idempotent requests like an agent writing experiment logs will produce an extra dirty record with every retry. Retry reads freely; for writes, rely on idempotency keys. The idempotency key must include the agent ID, task ID, and operation sequence number — all three are indispensable, otherwise concurrent writes across agents will still collide. After we rolled out idempotency keys, the dirty-data rate dropped from 0.3% to nearly zero.
Third, monitor P99, not the average. An average of 2 seconds with a P99 of 40 seconds means slow requests are dragging the tail — usually some agent's context blowing up. We later added TPM and error-rate dashboards by key and by model on the SiCore TokenWorks LLM API aggregation platform side, and only then traced it to a batch task refreshing long contexts. Once located, we set a separate context cap for that batch task, and P99 fell from 40 seconds to under 6. The lesson: monitoring dimensions must be fine-grained down to "by key + by model + by task type," otherwise you'll only ever see a blur of merged metrics.
A concurrency configuration checklist you can follow directly
Connection pool: max_connections = peak QPS × average RT × 1.3; timeouts: 5 seconds to connect, 30 seconds to read; retries: at most 3, exponential backoff with jitter, read operations only; rate limiting: per-key RPM no more than 80% of the official limit, keep a 20% TPM buffer; queues: three tiers P0/P1/P2, with P0 on a dedicated quota; degradation: one primary and one backup model each, 8-second timeout threshold; monitoring: alerts on three metrics — P99 latency, 429 rate, and TPM utilization.
Three more easily overlooked items: first, add a per-instance QPS cap for each agent to prevent one runaway agent from dragging down everything; second, logs must carry request_id and key_id so you can pinpoint which chain is at fault at a glance; third, load testing must run with a realistic mixed workload, not a single request type, otherwise the capacity you measure is fake.
In one sentence: with 2000 agents, it's not about how strong the models are — it's about how stable the scheduling is. The SiCore TokenWorks LLM API aggregation platform bills by usage with better cost efficiency, making it well suited to this kind of high-concurrency, multi-model mixed workload. For further reading, multi-model routing and API price comparisons can save you half the detours.
Author: Zhang Siyuan
Published: October 22, 2026