Background tools

Four tool patterns

Tool flags compose along four patterns. Pick the row that matches your tool and copy its flags; the flags are documented individually in the realtime reference.

The background job lifecycle

Declare the tool with deferred: true. When the model calls it, your host acknowledges the call with a defer frame instead of a result: from the browser SDK, client.sendToolDeferred(id, { handle, statusLabel }); from Python, send_tool_deferred(id, handle, status_label=...). That acknowledgement must arrive within the ordinary tool timeout; after it, the job's own clock applies: deferred_timeout, two hours by default and declarable up to 24.

Give every call its own handle, and check the acknowledgement. The handle identifies that call, not your worker or session, so re-using a live one is rejected: { accepted: false, reason: "handle_in_use" }. A rejected defer is not a deferral. That call stays on the ordinary tool timeout, so a host that ignores the ack ends up with a call it believes is backgrounded quietly expiring, taking the job with it. Mint a fresh handle per call (the call id is already unique, so <your-session>-<call-id> works well; the prefix matters, since a bare broker call id is reserved and rejected), and treat accepted: false as an error worth surfacing. Several calls can still feed one long-lived worker: keep your own handle-to-worker map; the broker never requires the handles to match. A handle is burned the moment its defer is accepted, so it is never reusable, even after that call finishes.

Undeferred calls are not on a stopwatch either: tool_progress and tool_partial_result re-arm the timeout (activity is liveness: only a fully silent window expires), and an open interaction suspends it, because a call waiting on the caller's decision is not a hung call. Both renewals stop after ten minutes of total call life, at which point the ordinary timeout fires. Anything genuinely longer belongs in a deferred call, whose declared deferred_timeout stays the ceiling and is never extended this way. A timeout that does fire is recorded in the session timeline as tool_deadline_timeout.

1. Hand-off. The agent voices the hand-off naturally ("kicking that off now") and the turn ends. The caller can change topic; the session stays fully conversational.
2. Progress. For anything longer than about thirty seconds, report progress as the job advances. There are two frames with different behaviour. A tool_progress note ("running the test suite") silently keeps the agent informed, so "how's it going?" always gets a current answer; it never speaks on its own. A tool_partial_result carries a structured segment of the eventual answer ("tests passed"), and with reply: true the broker narrates that milestone proactively, but only when the floor is free: it never interrupts the caller or an in-flight reply, and a skipped narration still lands in context. Curate accordingly: milestones speak, telemetry stays silent. Flag phase transitions (tests failed, blocked, done) for narration and send routine churn as silent notes, so the spoken interjections stay worth listening to.
3. Mid-call decisions. A job that discovers it needs the caller's decision (approve, disambiguate, supply a value) raises it with tool_partial_result plus interaction: {id?, prompt, options?}, never silently dropped: the broker asks by voice at the next opportunity and acks the lifecycle over tool_job_narration (with the interaction's stable id). If the decision gets made in your own UI first, or newer intent makes it moot, close it with tool_interaction_update (resolved/cancelled/superseded): pending or in-flight narration stops, the agent won't act on the stale ask, and the job itself keeps running. Every update gets a deterministic tool_interaction_update_ack; a user interruption of the ask's narration does not close the decision; it stays answerable and addressable until you close it or the call ends.
4. Completion. When the host reports the result, the broker voices it as a completion turn, with notify_on_complete, even if the conversation has moved elsewhere.
5. Interruption and cancellation. The caller can barge into any narration, and a server tool_cancel tells the host to stop pending work promptly, whether from timeout, discard, or an explicit request from the caller.
6. Reconnects. A dropped transport is not a dead job. Reconnecting with the latest resume token re-announces pending jobs to the host (tool_deferred_resume) within the broker's bounded resume window.

Spoken permissions

Declare requires_permission: true on any side-effecting tool and the call becomes a pending approval instead of executing. The agent asks; the broker then independently verifies that the caller's spoken reply is a clear, unconditional authorization of that exact action before the tool runs. Conditions, changed arguments, hypotheticals, quoted or reported speech, prompt-injection text, silence and ambiguity are never treated as approval. Unclear replies hold the request and the agent clarifies naturally. A clear refusal resolves the request as declined.

This composes with the patterns above: a confirmed action typically pairs requires_permission with expected_duration: "seconds", and a background job can require approval before it detaches. Cancellation is also conversational: the caller can cancel a pending or running job by saying so, without any client-side UI.

The failure contract

Long-running work needs explicit failure semantics, and they are part of the wire protocol rather than something to infer from silence:

• A completion voiceover that cannot be delivered is retried once, then reported as a terminal tool_job_delivery_failed frame. A completion turn that produces no audible speech counts as a failed attempt, never as "delivered".
• An injected turn that arrives while a reply is in flight and gets dropped is reported with an inject_rejected error frame, marked retryable unless the session has ended.
• Jobs cannot outlive their declared deferred_timeout; abandoned work is cancelled rather than left dangling in the session.

Exact frame shapes are in the AsyncAPI document and the realtime reference.

Reference implementation

Get an API key · Contact us