Run your first eval
- Start from the SDK. Install the Python SDK, then define a JSON case using the schema below.
- Write a case. A case is a JSON file: the agent's instructions and tools, the simulated user's goal, the opening message, the fixtures that answer tool calls, and the checks. Fields are listed below.
- Iterate in text. Start with a hosted text run; it is the fastest way to inspect behavior and tool fixtures. Run in voice once the behavior is right.
- Run it hosted. Push the case and start a run with the Python SDK (
EvalsClient), then follow it on the Evals dashboard. - Read the result. Transcript, tool calls with the fixture that answered each one, checks, and links to both session recordings.
Write a case
A case is a JSON document: name, starter, target (instructions, optional tools, voice, web_search, end_call, default true), simulator (instructions), fixtures, checks and limits (max_turns, timeout_s, silence_s).
| Field | What it controls | Good practice |
|---|---|---|
| Name | The label in run history. | Name the behavior and situation, such as "Cancellation after retention offer". |
| Target instructions | The role and policy being evaluated. | State the desired behavior and constraints, without scripting exact conversational wording. |
| Simulated user instructions | The goal, facts and temperament of the other participant. | Give the simulator enough context to act consistently and a clear condition for ending the conversation. |
| Opening user message | The first message sent to the target. | Make it a plausible request that actually activates the situation in the instructions. |
| Tools and fixtures | target.tools declares the client tools the target may call; fixtures answers them, as a fixed {"result": ...} or a field_store that records fields and reports what is still missing. | Declare every tool the production agent has. A call to an undeclared tool fails closed and is flagged on the run page. |
| Checks | contains, not_contains, regex, tool_called, fixture_complete, max_turns, and judge. | Combine a narrow deterministic check with one judge criterion; several checks that disagree make a case impossible to pass. |
| Limits | max_turns (default 20), timeout_s (600) and silence_s (30) bound the conversation. | Keep max_turns a little above the turns a good conversation needs, so a looping agent fails rather than runs on. |
A run keeps a copy of its cases, so editing a case later does not change past runs.
Checks
Use a required phrase for something exact, such as a mandatory disclosure. Use a judge criterion for an outcome that needs context, such as "The agent states the cancellation date and does not claim success before the tool result." Both can be combined.
Checks must agree with the case: a phrase that requires an action while the judge forbids it can never pass, and the opening message must create the situation the instructions describe. Read a failed transcript before changing a prompt.
Sample cases
Start with a multi-turn case that exercises at least one tool fixture and proves an outcome, not only exact wording.
Run a sample in text first, then in voice. A useful case has several turns and proves its tool actions, not only its wording; a one-turn exchange only proves the connection works.
Text and voice
| Mode | Use it for | What it exercises |
|---|---|---|
| Text | Fast behavioral iteration and deterministic regression checks. | Two ordinary Dialt text sessions, including the response model, instructions, tools and conversation state. |
| Voice | End-to-end validation after the behavior is stable. | Generated speech cross-piped between two sessions, including recognition, endpointing, turn-taking and speech output. |
Voice runs pass generated audio between the two sessions. Nothing plays on a speaker.
Read a run
A run has one result per case (times the number of repetitions). Each result shows status, modality, transcript, tool calls, checks, why the conversation ended, any error, and links to both session recordings.
| Status | Meaning |
|---|---|
queued | The run is saved and waiting to start. |
running | At least one case is executing. |
passed | Every configured check passed without a simulation error. |
failed | At least one result failed: a check failed, or a guard, timeout or connection problem stopped the conversation. |
error | Execution hit an infrastructure or provider error. |
cancelled | The run was cancelled before completion. |
The termination reason says why the conversation stopped. completed and max_turns are normal. completed means the agent ended the call with end_call; the call is listed with the other tool calls. silence_guard and repetition_guard stop a conversation that is going nowhere. simulator_ended and simulator_silent mean the simulated user ended the conversation or stopped answering; the checks still decide the result. Connection, timeout and infrastructure reasons are runtime failures, not agent behaviour.
Usage
Each result opens two Dialt sessions, the agent and the simulated user, billed to your account like any other session: the text rate for text, connected time for voice.
When a run fails
- Read the status, the termination reason and the error.
- Compare the transcript with each check.
- Check that the opening message actually creates the situation in the instructions.
- Open both session recordings for the audio and the full transcript.
- Retry only once you know whether the agent misbehaved or the run failed for a technical reason.
A queued or running run can be cancelled from its page.