Voice agents

Put an agent on phone calls and browser voice conversations, and build on those calls from your backend.

Why Use Voice Agents

Your agent already answers questions and takes actions in a chat. Now people want to call it: a patient booking an appointment, a customer checking on a delivery, a visitor of your site who would rather talk than type. Building voice from scratch means telephony webhooks, speech recognition, speech synthesis, interruption handling and a way to give the model tools, all of it in real time.

Squid gives an agent a voice with the same agent definition you already have. A browser conversation is one call:

Client code

const call = await squid.ai().voice().startWebCall({ agentId: 'front-desk' });
call.onTurn((turn) => console.log(`${turn.role}: ${turn.text}`));

A phone number is a connector and a few settings in Agent Studio, no code at all. And everything about a call reaches your backend: the transcript as it happens, the call records, incoming texts, and the call's context inside your AI functions.

Overview

A voice conversation is a normal agent conversation with a spoken front end. Someone speaks; the agent receives text; the agent's answer is spoken back. The agent keeps its instructions, knowledge bases, connectors, AI functions, memory and security rules. What changes is who does the hearing and speaking, the voice engine:

Engine	Where it runs
Twilio speech (Twilio connector)	Twilio transcribes the caller and reads the agent's streamed replies aloud; the agent answers every utterance.
OpenAI Voice (OpenAI Voice connector)	An OpenAI realtime model speaks and listens, and calls the agent as a tool whenever it needs knowledge or an action. Also serves browser calls.
ElevenLabs (ElevenLabs connector)	An ElevenLabs agent with ElevenLabs voices hosts the call and uses the Squid agent as its LLM.

The Voice connectors overview compares the engines. This page covers what you do in code.

When to use voice agents

Use Case	Recommendation
Answer and place phone calls, send and receive texts	A Twilio phone line on the agent
Talk to an agent from a web page, no phone number	startWebCall() with an OpenAI Voice connector
Record and analyze calls, notify staff, update a CRM	A voice session event handler
Let the agent recognize the caller, verify identity, book things	AI functions that use the call context
Turn an audio file into text, or text into an audio file	AI audio, not a live call
Voice input in a chat widget	The AI chat widget with enable-transcription

How it works

1. A call starts: Twilio rings the agent's number, your backend places a call, or a browser calls startWebCall().
2. Squid opens a voice session for the call: one conversation memory, the agent's instructions plus phone manners and the call facts, and the chat options the line or the request asked for.
3. Every completed utterance of the caller runs a turn of the agent (or, on the OpenAI engine, the realtime model runs a turn whenever it needs the agent). The reply is spoken.
4. Each milestone is emitted on the application's event bus: started, one turn per spoken turn of either side, ended. The Twilio connector records them; your handlers can too.
5. The call ends when a side hangs up, when the agent ends it, or when it is transferred.

The voice API is part of the TypeScript client and backend SDKs. From Python, phone lines need no code, and the connector operations are available through execute_function.

Quick Start

Five minutes to a voice conversation in the browser, before any phone number is involved.

Prerequisites

• A Squid application with an agent, such as the one from How to build an AI agent
• An OpenAI Voice connector on the application (an OpenAI project id and API key; no webhook is needed for browser calls)
• A backend project initialized with squid init, and a frontend that has a Squid client

Step 1: Allow users to talk to the agent

A browser call is subject to the same security rules as a chat with the agent. Allow signed-in users:

Backend code

import { secureAiAgent, SquidService } from '@squidcloud/backend';

export class FrontDeskSecurityService extends SquidService {
  @secureAiAgent('front-desk')
  allowSignedInUsers(): boolean {
    return this.isAuthenticated();
  }
}

Step 2: Start or deploy the backend

squid start

To deploy to the cloud, see deploying your backend.

Step 3: Start a call from the browser

Client code

const call = await squid.ai().voice().startWebCall({
  agentId: 'front-desk',
  greeting: 'Hi, this is the front desk. How can I help?',
});

// The caller's words as transcribed, and the agent's as spoken.
call.onTurn((turn) => appendToTranscript(turn.role, turn.text));
call.onEnd(() => showCallEnded());

// Buttons of your page.
muteButton.onclick = () => call.setMuted(true);
hangUpButton.onclick = () => call.stop();

The browser asks for microphone permission, connects to the OpenAI realtime model over WebRTC, and plays the agent's voice through an <audio> element the call creates (pass your own as sink). Squid runs the agent behind the call as the signed-in user, so the agent's tools and data rules see that user.

Step 4: Give the agent a phone number

Add a Twilio connector and, in Agent Studio, the Twilio ability to the agent with a number of the account. The phone line reuses the same OpenAI Voice connector, and a browser call then defaults to the line's voice and language.

Authentication and Configuration

Operation	Who may call it
startWebCall() and startOpenAiWebCall()	Application users under the agent's @secureAiAgent rules (or any user of a public agent), and backend code with the API key. The agent runs as the caller.
createSession() and answerOpenAiCall()	Backend code with the application API key only: they bind a phone call to an agent.
Connector operations (placeCall, sendSms, getCallTranscript, …)	Backend code with the API key; see using the Twilio connector from code.
Phone line settings (connectedIntegrations[].options)	Agent Studio, or backend code with the API key through setAgentOptionInPath(); see setting up the phone line from code.

Frontend code calls executables of your backend for anything that needs the API key, so the key never reaches the browser.

Where a browser call's settings come from

Every setting of startWebCall() is optional. Each one falls back, in order, to the agent's phone line when the agent has one with the OpenAI engine, then to the OpenAI Voice connector's defaults, then to Squid's defaults (gpt-realtime-2.1, the marin voice, en-US, one language for the whole call, a greeting spoken as written). integrationId is only needed when the agent has no such phone line and the application has more than one OpenAI Voice connector.

Core Concepts

The voice session

Every call is a voice session with a sessionId. All turns of a session share one conversation memory keyed by that id, so the agent keeps the call's context from the first word to the last, and the session id is on every event the call emits. On a phone call the session also carries externalCallId, the Twilio call SID, which is the key of the connector's call record.

Turns and transcripts

A turn is one completed utterance: { role: 'user' | 'agent', text, at }. The caller's turns are transcripts of speech; the agent's turns are what was spoken. Interrupted replies are cut where the caller interrupted. On the OpenAI engine the transcript covers the whole conversation, including the small talk the realtime model handles on its own.

What the agent is told on a call

Squid adds phone manners to the agent's instructions for every turn of a call: short spoken replies, no markdown or links, confirm names, dates and numbers by repeating them back, say dates the way a person would, end the call when the conversation is complete. It also adds the call facts: the caller's number, the number that was called, the current date and time, and the language to answer in. Your own instructions apply on top; you do not need to write phone manners yourself, only what the agent should do.

Opening the call

By default the line speaks its greeting as soon as the call is answered. With greetingMode: 'agent' the agent composes the opening line itself once the call connects: it is told to look the caller up first when it has a tool for that, so a returning caller hears "Welcome back, Dana" rather than a generic greeting, and the greeting text is what it says when there is nothing to personalize. The price is a moment of silence after pickup while that turn runs.

Languages

language (a BCP-47 tag such as es-MX) is the language the call starts in. On the OpenAI and ElevenLabs engines, languageMode decides whether the caller may switch: single keeps the whole call in that language, selected follows the caller into any of additionalLanguages, any follows the caller into any supported language. Switching is deliberate: a name or a single foreign word never changes the language, a whole sentence does.

The call context in AI functions

On the Twilio and OpenAI engines, every AI function the agent calls during a phone call receives the call in its agent context:

ctx.agentContext field	Description
twilioCallSid	The Twilio call SID: the id of the connector's call record and transcript.
twilioIntegrationId	The Twilio connector the call came through.
webCallerKey	For a browser call through Twilio, the key your backend gave the caller.

A browser call started with startWebCall() carries whatever you pass in chatOptions.agentContext, plus the caller's identity through the normal request context (this.getUserAuth() in a function).

Building on calls

Reacting to calls as they happen

Subscribe to AI_VOICE_SESSION_EVENT_TYPE with an event handler to follow every call live. This handler keeps its own transcript and, when the call ends, writes a summary for the staff:

Backend code

import { eventHandler, SquidService, TriggerEvent } from '@squidcloud/backend';
import { AI_VOICE_SESSION_EVENT_TYPE, AiVoiceSessionEvent } from '@squidcloud/client';

interface CallTurn {
  sessionId: string;
  role: 'user' | 'agent';
  text: string;
  at: string;
}

interface CallSummary {
  sessionId: string;
  agentId: string;
  callSid?: string;
  summary: string;
  endedAt: string;
}

export class CallLogService extends SquidService {
  private readonly turns = this.squid.collection<CallTurn>('call_turns');
  private readonly summaries = this.squid.collection<CallSummary>('call_summaries');

  @eventHandler<AiVoiceSessionEvent>(AI_VOICE_SESSION_EVENT_TYPE)
  async onVoiceSession(event: TriggerEvent<AiVoiceSessionEvent>): Promise<void> {
    const { kind, sessionId, agentId, externalCallId, turn, endReason } = event.payload;
    switch (kind) {
      case 'turn':
        if (!turn) return;
        // One document per turn, keyed by its moment: a redelivered event overwrites rather than duplicates.
        await this.turns.doc(`${sessionId}_${turn.at}_${turn.role}`).upsert({ sessionId, ...turn });
        return;
      case 'ended': {
        console.log(`Call ${sessionId} of ${agentId} ended: ${endReason}`);
        const turns = await this.turns.query().eq('sessionId', sessionId).sortBy('at').snapshot();
        if (turns.length === 0) return;
        const transcript = turns.map((doc) => `${doc.data.role}: ${doc.data.text}`).join('\n');
        // The built-in agent summarizes without any memory, so nothing of the call leaks into another chat.
        const summary = await this.squid
          .ai()
          .agent()
          .ask(`Summarize this phone call in three sentences for the staff, then list any follow-ups.\n\n${transcript}`, {
            memoryOptions: { memoryMode: 'none' },
          });
        await this.summaries.doc(sessionId).upsert({
          sessionId,
          agentId,
          callSid: externalCallId,
          summary,
          endedAt: new Date().toISOString(),
        });
        return;
      }
      case 'started':
        return;
    }
  }
}

The payload is an AiVoiceSessionEvent:

Field	Type	Description
kind	'started' | 'turn' | 'ended'	Which milestone this event reports.
sessionId	string	The voice session.
provider	'twilio' | 'openai'	Who runs the call: Twilio ConversationRelay, or the OpenAI Realtime API (phone and browser).
agentId	string	The agent on the call.
externalCallId	string, optional	The Twilio call SID on a phone call.
metadata	object, optional	Free-form data given when the call started; the connectors set channel (phone or web) and callSid.
turn	{ role, text, at }, on turn events	The completed turn.
endReason	string, on ended events	Why the session ended.

Events are delivered at least once and in no guaranteed order, which is why the handler above keys turns by their timestamp. Turn-based Twilio calls (voiceMode: 'gather') and calls on the ElevenLabs engine run outside a voice session and emit no events; the former are still recorded in the connector's collections.

Using the call context in AI functions

The most useful thing an agent can do on a call is recognize who is calling. This function reads the call record the Twilio connector keeps and looks the caller up by number; the agent calls it on its opening turn when the line uses greetingMode: 'agent':

Backend code

import { aiFunction, AiFunctionCallContext, SquidService } from '@squidcloud/backend';
import { TwilioCallRecord } from '@squidcloud/twilio-client';

interface CallAgentContext {
  twilioCallSid?: string;
  twilioIntegrationId?: string;
}

interface Patient {
  phone: string;
  firstName: string;
  lastName: string;
}

export class FrontDeskService extends SquidService {
  private readonly calls = this.squid.collection<TwilioCallRecord>('twilio_calls');
  private readonly patients = this.squid.collection<Patient>('patients');

  @aiFunction('Looks up whether the caller is a known patient, by the number they are calling from', [])
  async lookupCaller(_params: unknown, ctx: AiFunctionCallContext<unknown, CallAgentContext>): Promise<string> {
    const callSid = ctx.agentContext?.twilioCallSid;
    if (!callSid) return 'This conversation has no phone number. Ask the caller for the number on file.';
    const call = await this.calls.doc(callSid).snapshot();
    if (!call) return 'The call record is not available yet. Ask the caller for the number on file.';
    // The caller's number is the origin of an inbound call and the destination of an outbound one.
    const phone = call.direction === 'inbound' ? call.from : call.to;
    const [patient] = await this.patients.query().eq('phone', phone).limit(1).snapshot();
    if (!patient) return 'No patient has this number on file. Offer to register the caller.';
    // Only the first name: verify the caller before sharing anything else.
    return `A patient named ${patient.data.firstName} has this number. Greet them by name and verify before sharing details.`;
  }
}

Return short sentences that tell the agent what to do next, as above, rather than raw records. A function that shares personal details should insist on verification first, for example a code texted to the number on file; the phone assistant tutorial shows the whole flow.

Answering and sending texts

Incoming texts to the agent's number are SMS_EVENT_TYPE events, answered by the agent unless you take over; the agent and your backend send texts from the agent's number. See SMS on the Twilio connector page.

Calling out

Your backend places a call with placeCall({ agentId, to, instructions }); the agent greets whoever answers and pursues the purpose you gave it. See outbound calls.

Transfers and human handoff

The agent transfers a live call with its built-in transferTwilioCall function; tell it in its instructions when and to which number. When your application should take the call over instead, to join the caller to a staff conference for instance, claim it with the human handoff operation first, so the agent leaving the call does not hang the caller up.

Browser calls

squid.ai().voice().startWebCall(options) runs in the browser: it opens the microphone, produces a WebRTC offer, hands it to Squid, which creates the call on the OpenAI Realtime API with the agent's setup and answers with OpenAI's SDP, and connects the audio. Only the offer and the answer travel through Squid; the audio flows between the browser and OpenAI, while Squid steers the call from the side, running the agent whenever the realtime model asks and emitting the transcript events.

Client code

const call = await squid
  .ai()
  .voice()
  .startWebCall({
    agentId: 'front-desk',
    // Everything below is optional and defaults to the agent's phone line, then to the connector.
    voice: 'cedar',
    language: 'es-MX',
    languageMode: 'selected',
    additionalLanguages: ['en'],
    greeting: 'Hola, habla la recepción. ¿En qué puedo ayudarle?',
    instructions: 'The visitor is on the pricing page; help them pick a plan.',
    chatOptions: { agentContext: { pageUrl: window.location.href } },
    metadata: { source: 'pricing-page' },
    sink: document.querySelector('audio#agent-voice') as HTMLAudioElement,
  });

console.log(call.sessionId, call.callId);
call.onTurn((turn) => console.log(turn.role, turn.text));
call.onEvent((event) => {
  // Raw Realtime API events, for a talking indicator or debugging.
  if (event.type === 'response.done') console.log('The agent finished speaking');
});
call.onEnd(() => console.log('Over'));

Browsers only expose the microphone on https:// pages and on localhost. A custom WebRTC setup can call startOpenAiWebCall({ agentId, sdp }) with its own offer and receive { sessionId, callId, sdp } to set as the remote description.

Backend code holding the API key may broker a browser call too, with startOpenAiWebCall(): the page produces the offer, an executable of yours creates the call after checks of its own and returns the answer, and the page sets it as its remote description. The agent then runs with the API key rather than as the user.

API Reference

squid.ai().voice()

Method	Returns	Description
startWebCall(options)	Promise<AiWebVoiceCall>	Browser only. Opens the microphone, connects to the OpenAI realtime model and runs the agent behind the call.
startOpenAiWebCall(request)	Promise<StartOpenAiWebVoiceCallResponse>	Creates the call from a WebRTC offer you produced and returns OpenAI's answer, the call id and the session id.
createSession(request)	Promise<CreateAiVoiceSessionResponse>	API key. Opens a voice session for a Twilio ConversationRelay call and returns the wss:// relay URL to connect Twilio to. What the Twilio connector does for every call on the Twilio speech engine.
answerOpenAiCall(request)	Promise<AnswerOpenAiVoiceCallResponse>	API key. Answers a phone call that reached an OpenAI project's SIP endpoint with an agent. What the OpenAI Voice connector does for every phone call.

createSession and answerOpenAiCall are for applications that run their own Twilio webhooks or OpenAI webhook instead of the connectors. createSession takes the agentId, provider: 'twilio', optional externalCallId, chatOptions applied to every turn, metadata echoed on every event, greetingMode and greeting, connectTimeoutSeconds (how long the relay URL stays valid, default 5 minutes) and relayBaseUrl for a tunneled backend. answerOpenAiCall takes the agentId, the callId from OpenAI's realtime.call.incoming webhook, the integrationId of the OpenAI Voice connector, the realtime model's instructions, and optionally model, voice, allowInterruptions, interruptionSensitivity, greeting, greetingMode, externalCallId, chatOptions and metadata.

startWebCall() options

Option	Type	Description
agentId	string	The agent the realtime model asks for knowledge and actions. Required.
integrationId	string	The OpenAI Voice connector; needed only when the agent has no phone line on it and the application has several.
model, voice	string	The realtime model and its voice.
language	string	The language the call starts in, as an ISO code or BCP-47 tag.
languageMode	'single' | 'selected' | 'any'	Whether the caller may switch language mid-call.
additionalLanguages	string[]	The languages the caller may switch to in selected mode.
greeting	string	Spoken as soon as the call starts; without any greeting the model waits for the caller.
greetingMode	'phrase' | 'agent'	agent has the agent compose the opening line, with greeting as its fallback.
instructions	string	What this call is for, added to the realtime model's instructions.
allowInterruptions	boolean	Whether the caller's speech interrupts the model.
interruptionSensitivity	'low' | 'normal' | 'high'	How readily caller speech is detected.
chatOptions	AiChatOptions	Applied to every turn the agent runs: instructions, agentContext, functions, and so on.
metadata	object	Echoed on every event of the session.
sink	{ srcObject, autoplay, play() }	Where the voice plays; an <audio> element. Created on the page when omitted.

AiWebVoiceCall

Member	Description
sessionId	The voice session in Squid; its events carry it.
callId	The call at OpenAI.
onTurn(listener)	Receives every spoken turn, { role, text, at }. Returns a function that stops listening.
onEvent(listener)	Receives every Realtime API event of the call as sent over the data channel, { type, ...rest }.
onEnd(listener)	Runs once the call is over: the agent hung up, the connection dropped, or stop() was called.
setMuted(muted)	Silences or restores the microphone without ending the call.
stop()	Ends the call from the browser's side and releases the microphone.

Error Handling

Error	Cause	Solution
A web voice call needs a browser with WebRTC support	startWebCall() ran outside a browser, or in one without RTCPeerConnection and getUserMedia.	Call it from browser code on an https:// page or localhost.
NotAllowedError from the browser	The user denied microphone access, or the page is not served securely.	Ask for the microphone from a user gesture and serve the page over HTTPS.
The call is refused with a security error	The agent is not public and no @secureAiAgent rule allows the user, the same as a refused chat.	Add a rule, see securing AI agents.
AGENT_NOT_FOUND	No agent with that id exists in the application.	Check the agent id.
OPENAI_VOICE_INTEGRATION_REQUIRED	The agent has no phone line on the OpenAI engine and the application has zero or several OpenAI Voice connectors.	Add the connector, or pass integrationId.
OPENAI_VOICE_API_KEY_SECRET_NOT_FOUND	The connector's API key secret was deleted.	Save the connector again with the key.
API_KEY_REQUIRED	createSession() or answerOpenAiCall() was called without the application API key.	Call them from backend code.
OPENAI_CALLS_ARE_ANSWERED_NOT_RELAYED	createSession() was called with provider: 'openai'.	OpenAI calls are answered with answerOpenAiCall(); the relay is for Twilio.
The agent apologizes and asks to repeat	The agent's turn failed, for instance an AI function threw or the model provider returned an error.	Look at the application log; the caller is not left in silence, the failure is spoken.

Best Practices

1. Write instructions for the job, not for the phone. Squid adds the phone manners and the call facts on every turn. Describe the flow you want: greet, identify, verify, help, confirm, close.
2. Verify before you share. The caller's number is a hint, not proof of identity. Text a code to the number on file and have the agent confirm it before reading out personal details or booking, as in the phone assistant tutorial.
3. Keep call tools fast. A caller waits in silence while a function runs; on the OpenAI engine the model announces a wait past a moment. Prefer indexed lookups, and defer slow work to an event handler after the call.
4. Answer functions with a sentence for the agent. "No patient has this number; offer to register the caller" steers the conversation; a JSON dump does not.
5. Test in the browser first. startWebCall() exercises the same agent, tools and voice settings as the phone line, without a number and without paying for minutes.
6. Make event handlers idempotent. Events arrive at least once and unordered; key what you write by session id and turn timestamp.
7. Secure browser calls like chats. A public agent lets anyone on your page call it; a @secureAiAgent rule limits calls to the users you want, and the agent runs as that user.
8. Mind what the transcript holds. Call transcripts contain whatever callers say. Apply your data retention rules to twilio_call_transcripts and to anything your handlers store, and use prompt privacy rules where personal data must not reach the model.

See Also

• Voice connectors: Twilio, OpenAI Voice and ElevenLabs, and how to choose an engine
• Twilio connector: phone lines, SMS, outbound calls, records, browser calls and human handoff
• Build a phone assistant that books appointments: a complete example
• AI functions: give the agent tools
• Events: how event handlers work
• Securing AI agents: who may talk to the agent