The opening piece of this series named four tests that scale puts a voice agent through. The consistency piece took the first. This one takes the second: peak load. It's the test most pilots never sit, because a pilot rarely sees its busiest hour. Production sees it every month, usually on the day your customers most need to get through.
Why is peak load harder on a voice line?
Because a phone call can't wait quietly. A web page that loads a second late is annoying, and most people never notice. A voice agent that goes quiet for a second sounds like it didn't hear, and the customer says "hello?" over the top of the reply. People are that sensitive to timing in every language: a study of turn-taking across ten languages (Stivers et al., 2009) found speakers everywhere work to minimise silence between turns. A call also holds on to capacity for its whole length. It isn't one quick request. It's a live session that keeps speech recognition, the language model, and speech synthesis busy for minutes, turn after turn, and it can't be parked in a batch to finish later.
How far above the average is your peak?
Further than the monthly number suggests. Take an illustrative operation handling 1 million calls a month, with calls averaging three minutes. Spread evenly, that's about 70 calls at once. Now put 40,000 of those calls into the single busiest hour, and you need 2,000 at once, nearly thirty times the average. A deployment sized from the monthly total would look comfortable right up to the morning it fell over.
The peaks are also predictable, which means they're yours to find. In Southeast Asia, they follow the local calendar: gajian in Indonesia, kinsenas and katapusan in the Philippines, bill due dates, 11.11 and 12.12 sales, the run-up to Tết. Others you cause yourself. A payment reminder campaign to 200,000 customers can create its own inbound peak an hour later, when some of them call back to ask about the reminder, so outbound and inbound plans need to be read together. And then there are the peaks nobody schedules: a network outage or an app failure sends everyone to the hotline at once, and it's the hour your agent matters most.
What actually slows down under load?
Every step, and not by the same amount. Each turn passes through speech recognition, the language model, your business systems, and speech synthesis, and each one adds its own delay. Most of the extra delay at peak isn't slower computing. It's waiting. In LLM serving, time to first token includes the time a request spends queued before any work starts, and the Sarathi-Serve paper (Agrawal et al., OSDI 2024) shows tail latency climbing sharply with request load on a widely used serving system, with generation stalls that can last several seconds. Google's SRE book gives the plain arithmetic: when a full queue is ten times the number of workers, a task that takes 100 ms takes about 1.1 seconds, nearly all of it spent waiting.
The narrowest point sets your real capacity, and it may not be the AI. Model providers enforce limits on requests and tokens per minute. Phone lines and SIP channels run out. And your own CRM or billing API is often the first to struggle, since the same payday that fills the call queue also fills the app, the website, and the human agents' screens.
Why doesn't average latency tell you anything at peak?
Because slow moments compound, and customers experience the whole call, not the average turn. "The Tail at Scale" (Dean & Barroso, 2013) makes the point with servers: if each answers slowly just 1% of the time, a request that needs 100 of them is slow 63% of the time. A voice call works the same way. Say five steps per turn are each slow 1% of the time, independently. About 5% of turns are slow. Over a ten-turn call, about four in ten calls contain at least one slow turn. A dashboard can show a healthy average while a large share of customers hear a pause that sounds broken, and under real load those slow moments tend to cluster, because they share the same queues.
So measure per turn, from the end of the customer's speech to the start of the agent's audio, at the 95th and 99th percentiles, split by market and flow, during your actual peak hour rather than across the day. Then add the signals customers give you directly: how often they say "hello?" during a pause, how often they talk over the agent, and how often they hang up mid-silence.
What should happen to the call that arrives after capacity runs out?
Decide that before the day arrives, because the default is usually the worst option. When a system takes on more than it can finish, everyone gets slower. As the Amazon Builders' Library explains, the useful number isn't how much traffic a system receives but its goodput: the work completed quickly enough to matter. A good system's goodput levels off at full capacity. A bad one's collapses. For voice, that means making a few choices in advance:
- Protect the calls already in progress. Refuse new work early rather than letting every live conversation degrade. An honest wait message or a callback offer is better than 2,000 calls that all stutter.
- Rank what matters most. A card-fraud call shouldn't queue behind a loyalty-points balance check. Google's guidance on handling overload labels requests by criticality so the least important are dropped first, and the same idea applies to call flows.
- Plan a lighter mode. Under pressure, an agent can use shorter replies, cached answers to common questions, and pre-generated prompts, or route simple turns to a faster model. Exercise this mode regularly, since rarely used fallback paths tend to be the ones that break.
- Set an overflow path. Agree how many calls can move to human agents, and how the callback queue gets cleared afterwards, so overflow doesn't just move the outage to a different team.
- Pace outbound first. Outbound dialling is the one load you fully control. When inbound spikes on the same platform, slow the campaign before any customer who called you feels it.
- Cap the retries. Retries feel safe and are a classic way to turn a slowdown into an outage. In one example from Google, retries stacked across layers turn a single user action into 64 attempts on the database. Use retry budgets, and retry at one layer only.
How do you test for your peak before it happens?
Run a load test as you would a fire drill: on purpose, on your terms, and past the point where you expect things to break.
- Start from your real curve. Pull your busiest hours from last year's call data, then add growth and any campaigns planned for the same week.
- Test ramps and spikes. Gradual growth and a sudden surge fail differently, and new capacity starts cold, so scaling up in the middle of a spike may not arrive in time.
- Go past the limit, on purpose. You need to see what breaking looks like. A well-built system rejects the excess and keeps serving the rest; an untested one may fail for everyone at once.
- Include your real dependencies. A test that stubs out the CRM shows you the vendor's capacity, not yours. Agree a safe way to include your own systems, or simulate them responding at the speed they manage on a bad day.
- Measure from the phone line. Place real calls during the test and time them from the customer's side, with real accents and code-switched speech. Also check that quality holds: an agent that stays fast by giving shallower answers hasn't passed.
- Test the recovery. When the surge ends, does the system return to normal unaided, or does someone need to step in?
- Book it ahead of the date. Run it early enough to act on what you find, including any provider quota increases, which rarely happen on the day.
How does WIZ.AI handle peak load?
The platform runs at 100M+ AI calls a month, so peaks are part of normal operations rather than an exception. Talkbot offers elastic outbound capacity that can scale beyond 1M calls in 10 minutes, flexing up or down for deadlines, seasonal peaks, and market-by-market campaigns. Wizlynn handles inbound service with a hot transfer that passes the customer to a human agent with full call context, so a call that needs a person doesn't start again from zero. The Language Engine is tuned for speed, with real-time recognition under 100 ms and a time to first audio of 300 ms, and those are the kinds of numbers worth checking at your own peak rather than in a quiet demo. QM reviews every call, so if conversation quality slips during the busiest hour, it shows up in the same day's data, not only in next month's complaints.
What should you ask a vendor about peak load?
Five questions. What are your 95th- and 99th-percentile response times per turn at our peak concurrency, measured from the caller's side rather than per component? Where does your capacity run out first, and which limits, such as model quotas or phone lines, would we hit before you could scale? What exactly happens to the call that arrives after that point, and can we hear it? How long does new capacity take to come online during a spike? And will you run a load test with us, past the breaking point, ahead of our next peak? A vendor that has been through real peaks will have an answer for each, and probably a story about how they learned it.
The monthly average is a planning number. Customers only ever meet the hour they called in. Peak load is the test of whether that hour feels the same as any other. The next piece in this series moves to the third test, failure handling: what the customer hears when something in the chain breaks anyway, and how to hand them over cleanly. In the meantime, see how Talkbot flexes for campaign peaks, or book a demo and bring your busiest hour from last year.
References
- Universals and Cultural Variation in Turn-Taking in Conversation — Stivers et al., PNAS, 2009
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., OSDI 2024
- The Tail at Scale — Dean & Barroso, Communications of the ACM, 2013
- Addressing Cascading Failures — Site Reliability Engineering, Google
- Handling Overload — Site Reliability Engineering, Google
- Using Load Shedding to Avoid Overload — Amazon Builders' Library
- Rate Limits — OpenAI API Documentation
- Reliable at Scale: Why One Good Call Isn't Reliability — WIZ.AI
- Reliable at Scale: Same Question, Same Answer, Every Call — WIZ.AI
- Locally Fluent: Context Is What the Script Can't Translate — WIZ.AI
- TTS Latency in Real-Time Voice Agents — WIZ.AI
- Talkbot: elastic outbound voice AI — WIZ.AI
- Wizlynn: reliable multi-agent inbound service — WIZ.AI
- Language Engine: multilingual speech for real-time calls — WIZ.AI
- QM: compliance-driven quality monitoring for every call — WIZ.AI
Related insights
Book Demo