How to Build Evals for your Voice Agent
The most important benchmark in voice AI is the one you build yourself, for your own application. This post shows how to build it with Dialt Evals and a coding agent, and how to compare Dialt Fast and GPT Live on it.
1. Get an API key
Create an API key, make a temporary folder for the work, and save the key in a .env file there.
2. Give your coding agent one prompt
Start Claude Code or Codex in that folder and paste a prompt like this one. It points the agent at the Dialt Evals skill and describes the assistant you want to test.
Use https://dialt.com/docs/skills/dialt-evals/SKILL.md to evaluate an education enrollment agent that qualifies leads and books advisor calls. Cover successful booking, a prospect changing their answer, and booking failure. Compare Dialt Fast and GPT Live, then make one round of prompt and tool improvements and show the before-and-after results. Use my Dialt API key in .env and work in a temp folder.The agent writes the cases, the mock tools and the judge criteria, runs them, and reports back with links to each run. The prompt names three cases. You can ask for edge cases too.
3. Read the run page
Each run has a page that puts the models side by side, case by case. Voice is non-deterministic, so the run in the video repeats every case three times per model. For each model the page shows:
- whether each attempt passed, failed or was inconclusive, and how many criteria it met,
- the criterion score and the pass rate, with a confidence interval,
- whether the simulated caller followed its script,
- the response latency.
4. Open an attempt
Every attempt links to its session. There you can play the recording, read the transcript and the tool calls, and see the response time of each turn, measured by aligning the audio with the text after the call. Each session is also reviewed for its outcome, endpointing precision and recall, and tool-call accuracy.
5. Play the caller yourself
Each attempt also links to the playground, loaded with that case. The page gives you the caller's instructions and the opening line, and you speak the part. This is a good way to test with a real voice, to add background noise, or to check the case makes sense to a person.
Start in text mode with Dialt Smart
When you set up a new eval, run it first in text mode with a strong model such as Dialt Smart. On that strongest setting, a failure usually points to a bug in a case, a tool or the judge criteria. Once those are fixed, move to voice mode and a faster model such as Dialt Fast. The skill tells the agent to work in this order.
Keep improving the suite
All of your runs are stored on the Evals page. Have your coding agent read back through the transcripts and statistics and look for four things: errors made by the model, errors in the scoring rubric, criteria that are badly designed, and cases that are missing. Harder evals change the most between the first and second pass, because their tools and judge criteria need more work.
Today you can compare Gemini Live, GPT Live, Dialt Smart and Dialt Fast. Support for plugging in your own agent is coming soon.
For more on designing the evals themselves, see 3 Tips for Better Voice Agent Evals and the evals guide.