3 Tips for Better Voice Agent Evals

These are three tips for developers who test a voice agent by running it against a simulated caller.

A simulated caller tests your agent's tool calling and reasoning. It will not test everything. Background noise, and accents beyond those a simulated voice can produce, need real audio.

1. Test in text mode first

If your agent is a cascade of speech recognition, an LLM and text-to-speech, run only the LLM. This has two benefits:

• It is cheaper, because you are not running the full stack.
• It is faster than real time. A voice eval runs at the speed of the conversation, so a large suite takes much longer.

Many real-time voice models do not offer a text mode. Gemini Live and GPT Live always produce audio, so you have to transcribe their output to get text. Dialt supports text mode.

2. Use a strong caller

A simulation runs your assistant against a caller. If the caller makes a mistake, you cannot tell whether the assistant was wrong or the caller was.

A real-time voice model on the caller side still makes reasoning mistakes. Set the caller up as a cascade with a very strong LLM. The caller will respond more slowly, and that is a fair trade, because the caller has to be extremely accurate. Dialt's evals use a strong, slightly slower caller for this reason.

3. Use a strong judge

When the call is over, an LLM reads the transcript and scores it against your criteria. The judge has to be smart enough to spot the mistakes and to match what the agent did against what you asked for. Use a strong model here too.

Write success criteria and a rubric

Every eval needs criteria for what counts as a successful conversation, and a rubric the judge can follow to grade it.

To put these into practice, see How to Build Evals for your Voice Agent or the evals guide.