Conversation evals

Run your first eval

1. Start from the SDK. Install the Python SDK, then define a JSON case using the schema below.
2. Write a case. A case is a JSON file: the agent's instructions and tools, the simulated user's goal, the opening message, the fixtures that answer tool calls, and the checks. Fields are listed below.
3. Iterate in text. Start with a hosted text run; it is the fastest way to inspect behavior and tool fixtures. Run in voice once the behavior is right.
4. Run it hosted. Push the case and start a run with the Python SDK (EvalsClient), then follow it on the Evals dashboard.
5. Read the result. Transcript, tool calls with the fixture that answered each one, checks, and links to the available session recordings.

Build a suite with a coding agent

Give your coding agent the eval-building skill and describe your agent's purpose and tools. The skill guides case refinement and the transition from text validation to voice evaluation.

Use https://dialt.com/docs/skills/dialt-evals/SKILL.md to build an eval suite for my education enrollment agent. It qualifies leads and books advisor calls. Use mock CRM/calendar tools, then compare Dialt Fast and GPT Live in voice.

Organize cases into suites

A case describes one scenario and its criteria. A suite is a saved collection of cases. A run records one execution and its results.

Create cases and suites, and start runs with the Python SDK or API. A case can belong to several suites. Hosted runs use your account's credits.

Use the Evals dashboard to filter a suite's history, compare models and inspect individual results. Each run keeps the suite name and case contents from the time it started. Editing or deleting a suite leaves past results unchanged; deleting a suite also keeps its cases. Empty suites can be saved but need a case before they can run.

Eval API

Authenticate with your account API key using Authorization: Bearer dk_... (existing ck_... keys remain valid). Machine requests use https://api.dialt.com/v1; the paths below include that /v1 prefix.

• GET/POST /v1/evals/cases: list or create cases.
• GET/POST /v1/evals/suites: list or create suites; supply name and case_ids.
• GET /v1/evals/targets: available models, what each supports (capabilities) and the external credit reservation per attempt. dialt runs the target on Dialt Fast and dialt-smart on Dialt Smart; openai-live and gemini-live are external comparisons. The retired dialt-genius target can no longer be selected; runs that used it still load with that target_id.
• POST /v1/evals/runs: start a run with suite_id or case_ids, modality, repetitions and optional targets. Add "dry_run": true to check it first without creating anything.
• GET /v1/evals/runs and GET /v1/evals/runs/{id}: history and detailed results.

Collection responses use {results, next, has_more}. Requests accept an optional opaque signed cursor and an optional limit from 1 through 100 (default 50). Follow each returned next URL verbatim until it is null; never decode or construct a cursor. has_more is true exactly when next is non-null. Cursors are bound to the account, route, surface/origin, filters, ordering and limit; authorization is checked again on every request, and malformed or mismatched cursors return a structured 400 error. Cases, suites and runs are ordered by descending created_at, then UUID. Traversal is a live view rather than a snapshot; records created or deleted during traversal may change the collection. The SDK preserves its list-returning methods while following every continuation. Errors use {error: {code, message, details?}}.

Each target publishes the case settings it runs in capabilities: target_fields and tool_fields, the settings it ignored (accepted with a warning, such as web_search: false, voice or a tool's recorded_bridge on an external model; the value must still be valid), and limits. limits.timeout_s is the case conversation limit: at most 300 seconds for external models. It is separate from billing.cap_seconds, the billable ceiling per attempt, which adds the farewell drain and bounded shutdown. A run request with "dry_run": true checks every case against every selected target. It returns 400 with every problem in error.details.errors (each has target, case, field and message; target, case or field is null when the problem is not specific to one), or 200 with the plan: its warnings, attempt count, the max_external_charge_micros to accept and whether your credit covers the run. The plan also returns valid, modality, execution, repetitions, targets, billing and credit (required_micros, sufficient; null for Dialt-only runs). It creates no run, reserves no credit and records no idempotency key. A created run keeps the same warnings.

For external targets, accept max_external_charge_micros: the sum of their per-attempt reservations times the case count and repetitions. The dry run returns this value. Unused credit is returned. The Python SDK's start_run takes the same targets, dry_run, max_external_charge_micros, replace_unassessable and max_replacements.

Set "replace_unassessable": true so each scheduled attempt (case, target and repetition) ends with an assessable result. An attempt that produced no evidence about the model, because of an infrastructure error, an eval worker restart, a harness or simulated caller connection failure, a simulated caller that broke its script, or a simulated caller that heard the model and stopped responding, gets one fresh replacement attempt in the same run. A passed or failed attempt, a failure on the model's side (including a model that never opened the call), a failure with no established cause and an attempt with a failed check are never replaced. The replaced attempt keeps its transcript and returns replacement_decision (for example the label "Replaced: simulated caller broke script") and counts_in_slot: false; the replacement returns replacement: 1 and replaces_attempt_id. Run status, pass/fail and criterion_scores count the replacement; caller adherence and latency count every conversation. max_replacements caps replacements across the run (default a quarter of the attempts, rounded up). Replacements cost like any attempt, so the option is off by default: for external models, max_external_charge_micros includes one reservation per external replacement the budget allows, and the dry run quotes it. A replacement that would exceed that charge or your credit is skipped with its reason. Each run returns replacements with the replacements made, those skipped by reason and the attempts spent.

For example, compare an existing suite using requests. Set DIALT_API_KEY and DIALT_EVAL_SUITE_ID in your environment:

import os
import requests

api = "https://api.dialt.com/v1/evals"
client = requests.Session()
client.headers["Authorization"] = "Bearer " + os.environ["DIALT_API_KEY"]

run = {
    "suite_id": os.environ["DIALT_EVAL_SUITE_ID"], "modality": "voice",
    "targets": ["dialt", "openai-live", "gemini-live"],  # add "dialt-smart" to compare Dialt Smart
    "repetitions": 1,
}
plan = client.post(api + "/runs", json={**run, "dry_run": True}, timeout=30)
if plan.status_code == 400:
    details = plan.json()["error"].get("details", {})
    for problem in details.get("errors", []):
        print(problem["message"])
    print(details.get("reason", ""))
plan.raise_for_status()
for warning in plan.json()["warnings"]:
    print("ignored:", warning["message"])
response = client.post(api + "/runs", json={
    **run, "max_external_charge_micros": plan.json()["max_external_charge_micros"],
}, timeout=30)
response.raise_for_status()
print("https://dialt.com/evals/" + response.json()["id"])

Write a case

A case is a JSON document: name, starter, target and simulator (each a session mode document exactly as a session start carries it: instructions, greeting, tools, voice, web_search, end_call, and every other session option; for the target end_call defaults to true, and the simulated user's tools, end_call and greeting are set by the run), fixtures, checks and limits (max_turns, timeout_s, silence_s).

A run keeps a copy of its cases, so editing a case later does not change past runs.

Talk to a case

Every case response includes playground_url, and so does each run attempt (null once its case is deleted). Open it while signed in with the case's owner (personal or organization) selected to talk to the case's target in the playground: you play the simulated user, whose instructions and opening line are shown on the page, and the case's fixtures answer the target's tool calls. Use it to hear an agent before wiring up your own tool host, to check that a new case's scenario is realistic, or to reproduce a failed attempt yourself. The conversation is an ordinary playground session in Sessions; it is not an eval attempt and is not scored as part of a run.

Align tools, fixtures and checks

The tool tells the agent what it can do. Its fixture supplies evidence of what happened. A check assesses that declared contract; it must not introduce a rule the agent never received.

For example, describe request_callback(phone) as: "Submit a callback request once the caller supplies a phone number. Success records the request; it does not book a time or guarantee a callback." Describe the phone format in its parameter schema. Return {"result": {"status": "requested", "request_id": "cb-1"}}.

Use {"type": "tool_called", "value": "request_callback", "args": {"phone": "+15555550123"}} to check submission. Separately judge: "After the request receipt, the agent may confirm submission but must not promise contact or a booked time." "Your request was submitted" meets that contract; "Someone will call you tomorrow" does not, even if submission succeeded.

State prerequisites, optional-field meaning, partial submission rules, effects and result guarantees in each tool declaration. Use fixture_complete for required stored fields. Reserve judges for meaning or required ordering, and do not restate a deterministic check as a judge: both count in the criterion score, so the overlap counts one requirement twice. Set a weight to make a requirement count more; give each criterion an observable trigger and outcome, and separate caller coverage from target behavior. Specify whether a requirement applies throughout the conversation or at a particular step. If confirmation is required, make the invitation to confirm explicit in both the agent instructions and the criterion. When a result surprises you, first compare the declaration, fixture and criterion, then inspect the agent's actions.

Checks

Use a required phrase for something exact, such as a mandatory disclosure. Use a judge criterion for an outcome that needs context, such as "The agent states the cancellation date and does not claim success before the tool result." Both can be combined.

Checks must agree with the case: a phrase that requires an action while the judge forbids it can never pass, and the opening message must create the situation the instructions describe. Read a failed transcript before changing a prompt.

Start with a runnable simulation case that exercises tool fixtures and checks an outcome. Run it in text first, then in voice.

Text and voice

Voice runs pass generated audio between the two sessions. Nothing plays on a speaker.

External models run in text and voice. Neither Live model can answer in text, so in a text run each receives the caller's turn as text and its reply is the transcript of the audio it generated, which is discarded. Both receive the turn as a typed user message and speak the answer. Their results are labelled (text: transcribed audio output). Each target's modalities lists the run modalities it accepts.

Read a run

Total conversation time measures opening dispatch after both connections are ready through termination and queued speech drain. It includes tool waits and silence, but excludes connection setup, provider finalization and grading. The API returns conversation_duration_s; older attempts without a measurement return null. Compare the same cases and modality alongside pass rate: a short failed or prematurely ended conversation is not an efficiency win. This is conversation duration, not speech response latency.

Response latency in voice runs is the time from the caller's last word ending to the model's first word starting, measured by forced alignment of a Dialt session recording. For a Dialt model, response_latency uses the assistant's own session recording: the same figure as that session's aligned_response_ms, on its recording clock. OpenAI Live and Gemini Live have no Dialt recording, so their response_latency uses the simulated caller's recording: the caller's speech and the model's reply as the caller heard it, on the caller's recording clock. caller_response_latency measures every model that way, so compare Dialt and external models there; it includes the relay between the two sessions. Neither is a real caller's device latency. Each latency object names its measurement and returns state, n, p50, p90 and per-turn gaps in milliseconds; each run pools turns per model in targets. Alignment finishes after the conversation, so recent results report pending first. Text runs report unavailable with a reason, never zero. case_snapshot_sha256 identifies the cases, so you compare latency and scores only across runs of the same cases, targets and brain tiers. A Dialt target shows its tier (Dialt Fast or Dialt Smart), not the models that serve it.

A run has one result per case, repetition and selected model. Each result shows status, modality, transcript, tool calls, checks, why the conversation ended, any error, and links to the available session recordings.

Criterion score is the weighted share of criteria met: every judge criterion and every deterministic check (contains, tool_called, fixture_complete and the rest). Each attempt returns criterion_score with passed, failed and unknown counts. With no unknowns, score is the fraction met; otherwise score is null and lower and upper bound it, with coverage the assessed share. Each run returns criterion_scores, one per model: repetitions are averaged within a case, then every scheduled case counts equally. An attempt that could not be judged keeps its criteria as unknown, so the range widens instead of the case disappearing. A check never passes on missing evidence: when the agent never spoke or called a tool, or the conversation was cut short or inconclusive, a phrase that was said or a tool that was called still fails a not_contains or tool_not_called check, and every verdict that rests on absence is unknown. Cases without criteria are reported as unscored. caller_probe assesses the simulated caller and counts only in pass/fail. Any other check may set a weight above 0 and at most 1000 (default 1). Each score reports its scoring_version; version 2 added deterministic checks, and every run is scored by the current version when read.

Pass rate adds sampling uncertainty to strict outcomes: 3 passes in 3 attempts is weak evidence. Each run returns pass_rates, one per model, with attempts, assessed (passed or failed), passed and unknown (inconclusive, error, cancelled or unfinished) counts, pass_rate over assessed attempts and a 95% Wilson interval in pass_rate_lower and pass_rate_upper. by_case gives the same for each case: 3 of 3 passes reads 44 to 100%. by_case is returned on the run detail only. The model's rate weights every case equally, like the criterion score, and its interval uses the effective sample size of that weighting. It covers only cases with an assessed attempt (assessed_cases), so compare models on that count too, and read unknown beside it: a dropped connection is unknown, whichever side dropped it.

Caller adherence reports whether the simulated caller followed its script. Each judged hosted attempt of a case with caller instructions carries one simulated caller coverage verdict (a judges entry of type simulator_adherence). When the case opens with target.greeting, the attempt also carries a second simulator_adherence verdict, "Simulated caller role after the target opens", which checks that the caller replies as the person its instructions describe; the caller followed its script only when every such verdict passes. A verdict with opportunity: false means the model ended the call before the caller's instruction applied; it is left out, and an attempt with no other verdict is not counted. An attempt cut short by an error, cancellation, stall or timeout counts only when that verdict is established, except a stall on the simulated caller's side (it heard the model and stopped responding), which counts in full. Each run returns caller_adherence, one entry per model, with assessed, followed, broke_script and unknown attempt counts and rate, followed over assessed. An attempt where the caller broke its script, or where that was not established, is inconclusive, so a low rate means fewer of your attempts were scored. The run page shows it as "Followed its script in 11 of 12 attempts".

The termination reason explains why the conversation stopped. completed means the target requested an end; simulator_ended means the simulated caller did. stalled means neither side spoke for silence_s. When the evidence shows which side owed the next reply, the attempt's ending.failure.side names it: target when the model recognized the caller's turn and never replied, or stopped mid-sentence, and simulator when the simulated caller heard the model and never replied. A stall on the model's side is a failed attempt and counts in pass rates. Any other stall, including one after a reconnect, an error, a pending tool call or speech the other side never recognized, and every timeout, is inconclusive. timeout_s bounds the conversation: an end already under way when it passes finishes as completed or simulator_ended, so timeout means neither side ended the call in time. target_did_not_open means a model given target.greeting did not start speaking its greeting within 8 seconds; the attempt is failed. Invalid caller coverage or an unavailable judge also makes an attempt inconclusive. Inspect the individual evidence before drawing a model conclusion.

Each hosted attempt also returns ending, which reports the parts of the end separately: conversation (whether it ended normally, who ended it and how long it ran), drain (whether the final speech was delivered within its grace window), shutdown (whether an external model's connection closed cleanly), failure (the failure code, the side and component that failed, and the phase), recovery (an attempt interrupted by an eval worker restart, and whether it runs again) and last_activity (when each side last spoke, last recognized speech and last received audio). Their values are fixed codes: conversation.outcome is ended, stopped or cancelled; drain is drained, grace_expired, limit_reached or cancelled; shutdown is closed or failed; for infrastructure_error, failure.cause is the public error code such as insufficient_credit, or null. ending is null for local runs and for attempts that have not run, except a queued attempt waiting after a worker restart, which carries only recovery. For an external model, an ending near the case limit is cut short so its farewell finishes within billing.cap_seconds. A hosted attempt's error is the failure code (ending.failure.code), evaluation_failed when an error has no classified failure, and empty when there is none. Every attempt carries a correlation_id; include it when you report a problem so Dialt can find the attempt in its logs.

Comparison runs display completion separately from each model's outcomes. For compatibility, the API run status can be failed when only coverage is incomplete; outcome="inconclusive" distinguishes this from scored failure.

Usage

Each attempt bills two sessions, rounded to connected seconds. The target bills at its model's rate: dialt at the fast rate, $0.065 per minute, dialt-smart at the smart rate, $0.085 per minute, and external models at $0.065 per minute. The simulated caller bills separately at the smart rate, $0.085 per minute, so a one-minute voice attempt against dialt costs about $0.15. External models reserve up to billing.cap_seconds per attempt (330 seconds, $0.3575); unused credit is returned, and external infrastructure failures are waived. Text runs bill Dialt sessions at the ordinary Dialt text rate; an external model in a text run still bills its connected seconds at $0.065 per minute, capped at the same reservation.

Safe retries

POST /v1/evals/cases, POST /v1/evals/suites and POST /v1/evals/runs accept an optional Idempotency-Key header. Runs also accept an idempotency_key body field; if both are supplied they must match. Keys are 1 to 128 visible ASCII characters without spaces and are scoped to the authenticated account. Reuse a key with identical JSON request content after a lost response to receive the original case, suite or run (201) without a duplicate, and for a run without duplicate execution or credit reservation. Object-key order does not matter; array order and explicit versus omitted defaults do. Different content with the same key returns 409. Keys remain reserved for the resource's lifetime; replaying a case or suite key after deleting that resource returns 409, and after replacing it with PUT returns its current content. Without a key, each request creates new work.

Case and suite names are unique per account. Creating or renaming one to a name already in use returns 409 with the existing resource's ID in error.details.existing_id, so a script can adopt it or rename.

A repeated local report key must carry the same normalized evidence. Changed evidence returns 409; it no longer silently returns the earlier report.

Public API responses expose Dialt tiers, including for team-role requests. Hosted target_config contains the customer's frozen target input, not the private provider execution configuration. Platform-owned target profiles, judge results, usage and events are projected to customer fields; exact model identities remain in authorized internal diagnostics. Customer-authored local reports and conversation content retain their own content.

An API key sent to the dashboard origin returns 401 with a message naming the API origin to use.

When a run fails

1. Read the status, the termination reason, the error and ending: a failure names the side and component that failed, and last_activity shows which side went quiet.
2. Compare the transcript with each check.
3. Check that the opening message actually creates the situation in the instructions.
4. Open both session recordings for the audio and the full transcript.
5. Retry only once you know whether the agent misbehaved or the run failed for a technical reason.

A queued or running run can be cancelled from its page or with evals.cancel_run(run_id). To delete a finished run and its results, use evals.delete_run(run_id) or DELETE /v1/evals/runs/{id}. Deletion returns 204; active runs return 409, so cancel and wait for completion first. Missing or already deleted runs return 404. Cases, suites, session recordings and billing history are preserved. Reusing a deleted run's creation idempotency key returns 409.

View sessions · Open evals