The opening piece of this series named four tests that scale puts a voice agent through: consistency, peak load, failure handling, and control. This piece takes the first. It's the quietest of the four. An inconsistent agent doesn't crash, go silent, or set off an alert. It just gives one customer one answer and the next customer another, and each call looks fine when you review it alone.
What does consistency actually mean for a voice agent?
Not identical wording. A voice agent that repeats the same sentence word for word to every caller sounds like an old IVR, and natural phrasing is part of why customers stay on the line. Consistency means the parts that matter don't move: the decision (eligible or not, approved or escalated), the facts (the fee, the deadline, the balance), the policy (what the agent may offer and what it must disclose), and the action it records in your systems. So the test is simple to state. Same question, same customer situation, same policy should give the same outcome, whether it's the first call or the hundred-thousandth, a Monday morning or a Saturday night, before last week's update or after it.
Why doesn't the same agent give the same answer every time?
Partly because of the model, and partly because of the call. On the model side, large language models generate text by sampling, and even settings that should make them repeatable often don't. In a study of five LLMs configured to be deterministic (Atil et al., 2024), accuracy on the same tasks varied by up to 15% across runs, and none of the models delivered repeatable accuracy across every task. When Thinking Machines sent one prompt to a model 1,000 times at temperature zero, it got 80 different completions back. They traced much of that to the server: how many other requests happen to be processed alongside yours changes the arithmetic slightly. In other words, output can shift with load, which is exactly what changes at scale.
On the call side, the input is never truly identical. Two customers asking the same thing phrase it differently, call from a quiet office or a busy market, and speak with different accents or switch languages mid-sentence. Speech recognition turns each into a slightly different transcript. The knowledge search may return a different passage. Each difference is small. Stacked together, they're enough to tip a borderline answer one way on one call and the other way on the next.
Where does inconsistency do the most damage?
Wherever the answer carries a commitment. In banking and insurance that's fees, waivers, cooling-off periods, and claim eligibility. In collections it's the payment plan offered, the deadline quoted, and the consequences described — and if two customers in identical arrears are offered different terms, that's a fairness problem as well as a service one. The legal side is no longer theoretical. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline responsible for its website chatbot's incorrect description of its bereavement-fare policy and rejected the argument that the chatbot was responsible for its own statements. Whatever your agent tells customers, at whatever volume, counts as what your company told them. Inconsistency is also the hardest of these to catch by sampling. If 1 call in 200 gets the wrong version of a policy, a reviewer listening to 2% of calls will hear it about once a week and may well put it down to a one-off.
How do you measure consistency?
Run the same thing many times and count how often the agent gets it right every time. That's the idea behind the pass^k measure in τ-bench (Yao et al., 2024), mentioned in the opener: success on all k attempts, not just one. In practice:
- Build a golden set of policy-critical questions. Pick the 50 to 200 situations where a wrong or wavering answer costs money or creates compliance exposure: eligibility checks, fee and deadline questions, required disclosures, refusals. Agree on the correct outcome for each with your compliance team.
- Vary the call, not the case. For each situation, test several phrasings, accents, noise levels, and code-switched versions, drawn from your own recordings where you can. This is the same approach as the Locally Fluent test plan, pointed at stability instead of coverage.
- Repeat each one. Ten to twenty runs per variant is enough to see whether an answer is solid or just usually right.
- Score the outcome, not the sentence. Extract the decision, the figures, and the action logged, and check that they agree across runs. Atil et al. call this the total agreement rate of parsed answers. Different wording around the same outcome passes. Same wording around a different number fails.
- Report it per segment. A 99% every-time rate overall can hide 90% in one language or one product line, and that segment is where the complaints will come from.
How do you stop updates from changing answers that used to be right?
Treat every change as a release, because every change can move answers: a new model version, a revised prompt, an edited knowledge article, a new scenario. Model updates alone can shift behaviour a lot. When researchers compared the March and June 2023 versions of GPT-4 (Chen, Zaharia & Zou), its accuracy on one simple task fell from 84% to 51%, under the same product name. Three habits keep this under control. Pin the model version, so nothing changes underneath you without a decision. Run the golden set as a regression suite before every release, and block the release if any policy-critical answer changes without a matching policy change. Then release gradually: send the update to a small slice of live traffic first, compare its outcomes with the current version, and only then roll it out wider — the canary release practice software teams use for any risky change. Keep the previous version ready, so rolling back takes minutes, not a project.
How do you design an agent to be consistent in the first place?
Testing finds inconsistency. Design keeps it from appearing. The main principle is to let the language model handle the conversation and keep the decisions out of it. Facts like balances, fees, and due dates should come from your systems of record through a lookup, not from the model's memory of a document. Eligibility rules should run as rules, with the model passing in the customer's details and explaining the result. Regulated disclosures can use approved wording, delivered naturally but not paraphrased into something new. Answers to policy questions should be grounded in an approved knowledge base, and when the agent can't find a confident answer it should say so and escalate rather than improvise. Guardrails set the outer limits on what the agent can say and do. Every piece of a decision that moves out of free generation and into a deterministic step is one less thing that can change from call to call.
How does WIZ.AI keep answers consistent at scale?
Consistency is built into each stage, from design to monitoring. Wizlynn answers from a knowledge base of verified service information and only within approved policies, with guardrail controls on its generative responses and guarded escalation when a question falls outside them. Its multi-agent design gives each agent a narrow, well-defined job, which leaves less room for answers to drift. Changes go through the AI ops loop, which checks each update against real calls before release, so a fix for one scenario doesn't quietly change the answer in another. And QM reviews every call rather than a sample, so if two customers in the same situation start getting different outcomes, that pattern shows up in the same day's data instead of in a complaint a month later.
What should you ask a vendor about consistency?
Four questions. Will you run our policy-critical scenarios many times over, with our own phrasings and accents, and show us how often the agent gets each one right every time? Which parts of the answer come from a deterministic lookup or rule, and which are generated? What happens to our answers when you update the model or the platform — do we get a regression report before the change reaches our customers, and can we roll it back? And how would we find out, in live traffic, if the same question started getting two different answers? A vendor that has thought about consistency will answer with a process and data, not with "our model is very accurate."
An agent that's right on average can still be wrong for a thousand customers a month, and none of them will care about the average. Consistency is the first test of reliability because it's the one customers notice when they compare notes. The next piece in this series moves to the second test, peak load: what happens to response time and capacity on the busiest hour of the busiest day. In the meantime, see how Wizlynn keeps answers inside approved policy, or book a demo and bring the ten questions your customers ask most often.
References
- Non-Determinism of "Deterministic" LLM Settings — Atil et al., 2024
- Defeating Nondeterminism in LLM Inference — Horace He, Thinking Machines Lab, 2025
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao et al., 2024
- How Is ChatGPT's Behavior Changing over Time? — Chen, Zaharia & Zou, 2023
- Canarying Releases — The Site Reliability Workbook, Google
- Moffatt v. Air Canada, 2024 BCCRT 149 — Civil Resolution Tribunal of British Columbia
- Reliable at Scale: Why One Good Call Isn't Reliability — WIZ.AI
- Locally Fluent: Test It on Your Calls, Not the Demo — WIZ.AI
- Fast to Deploy Doesn't End at Go-Live: Inside the AI-Ops Loop That Keeps Updates Safe — WIZ.AI
- Agent Guardrails for Voice AI Customer Operations — WIZ.AI
- Wizlynn: reliable multi-agent inbound service — WIZ.AI
- QM: compliance-driven quality monitoring for every call — WIZ.AI
Related insights
Book Demo