Skip to main content

OpenAI Voice

Answer phone calls and browser calls with an OpenAI realtime model that has a Squid agent behind it​

The OpenAI Voice connector's functionality​

The OpenAI Voice connector puts an OpenAI Realtime speech-to-speech model on the line:

  • Natural conversation. The realtime model hears and speaks directly, with no transcription step in between: it handles greetings, small talk and confirmations itself, and can be interrupted.
  • The Squid agent as its brain. Whenever the caller asks for information or wants something done, the model calls the agent through an ask_agent tool. The agent answers with its knowledge bases, connectors and AI functions, and the model relays the answer in its own voice.
  • Phone calls, as the voice engine of a Twilio phone line.
  • Browser calls straight from a web page over WebRTC, with no phone number involved, through squid.ai().voice().startWebCall().
  • Languages. The call starts in the line's language and can follow the caller into selected languages, or any language.
  • Transcripts. Both sides of the call are transcribed and recorded like on every other engine.

Audio flows between the caller and OpenAI. Squid only steers the call: it sets it up with the agent's instructions, runs the agent when the model asks, and hangs up when the model ends the call.

Before you begin​

  • An OpenAI organization with access to the Realtime API.
  • The project the calls should run in and its id (proj_…), shown in the project's settings. Phone calls reach OpenAI through the project's SIP endpoint, sip:<projectId>@sip.api.openai.com.
  • An API key created in that project.
  • For phone calls, a webhook endpoint registered in the project, described below. Browser calls need no webhook.

Adding the OpenAI Voice connector to your Squid application​

  1. Navigate to the Connectors tab in the Squid Console.

  2. Click Available Connectors.

  3. Find the OpenAI Voice connector, and select Add Connector.

  4. Provide the following configuration details:

Connector ID: A string that uniquely identifies the connector in your code, such as openai_voice.

API Key: An API key of the OpenAI project. Stored as a secret.

Project ID: The OpenAI project id (proj_…).

Webhook Secret: Optional but recommended. The signing secret (whsec_…) of the webhook endpoint you register in the project. With it, Squid verifies that incoming-call webhooks come from OpenAI; without it, any request to the webhook URL is trusted.

Default Model: Optional. The realtime model for agents that pick none. Defaults to the newest realtime model, gpt-realtime-2.1.

Default Voice: Optional. The voice for agents that pick none. Defaults to marin.

The Advanced section holds Webhook Base URL, the public base URL OpenAI reaches this application at (only needed behind a proxy, a tunnel or a custom domain), and API Base URL, OpenAI's API base URL (default https://api.openai.com).

The connector configures nothing per agent. Which agents answer with it, with which model and voice, is decided on each agent's phone line.

Using OpenAI Voice on a phone line​

  1. Add the Twilio connector to the application, if you have not yet.

  2. In Agent Studio, add the Twilio ability to the agent (or open it), and pick a Phone number.

  3. Set the Voice engine to OpenAI voice and choose the OpenAI Voice connector to use.

  4. Choose the OpenAI model (the account's realtime models, newest first) and the OpenAI voice. marin and cedar are the newest voices; alloy, ash, ballad, coral, echo, sage, shimmer and verse are also available.

  5. Set the Language the call starts in and, under Languages, whether the caller may switch to selected languages or to any language.

  6. Write the Greeting and choose the Opening line: the greeting as written, or a line the agent composes once the call connects, so a known caller is greeted by name.

  7. Save. The status line reports that the number rings this agent, and shows the webhook URL to register in the OpenAI project. Register it as described next, once per connector.

The full list of options, including allowInterruptions and interruptionSensitivity which are set from code, is in the Twilio connector's phone line settings.

Registering the incoming-call webhook​

OpenAI announces every call that reaches the project's SIP endpoint by posting a realtime.call.incoming event to a webhook of the project. Squid answers that call from the webhook, so the webhook must point at your application:

  1. Get the webhook URL: it is shown in the Twilio ability's status line after saving, and in provisioning.openAi.webhookUrl on the agent's phone line. It looks like https://[APP_ID].[REGION].squid.cloud/webhooks/openAiVoiceIncoming?integrationId=openai_voice. From code, sipTarget() on the client returns it together with the SIP URI.

  2. In the OpenAI platform, open the project's Settings > Webhooks and create an endpoint with that URL, subscribed to the realtime.call.incoming event.

  3. Copy the endpoint's signing secret and save it as the connector's Webhook Secret.

One endpoint serves every agent that uses the connector: the call's SIP headers name the agent. A project can only announce calls to the endpoints registered in it, so a second application using the same project needs its own endpoint, and a development and a production application using one project each need theirs.

Local development

The webhook URL of a backend started with squid start includes your developer id and is public, so a development endpoint can point at it. Keep in mind that every endpoint of the project receives every call: a call for an agent your development application does not have is rejected by it, and answered by the application whose agent it is.

How a phone call flows​

  1. Twilio receives the call on the agent's number and asks the application how to handle it. The Twilio connector bridges the call to the OpenAI project's SIP endpoint, with SIP headers naming the agent, the Twilio call and the Twilio connector.
  2. OpenAI posts realtime.call.incoming to the registered webhook. The OpenAI Voice connector verifies the signature, reads the agent from the headers and checks that the agent's phone line uses this connector; a call it does not recognize is rejected.
  3. Squid accepts the call with the line's model, voice and language rules, instructions that describe the Squid agent, and two tools: ask_agent and end_call. It speaks the greeting, or has the agent compose the opening line.
  4. During the call, every ask_agent call runs one turn of the Squid agent, with the call's context and memory, and hands the answer back to the model to speak. Caller and agent turns are recorded as they are transcribed.
  5. When the caller says goodbye, the model calls end_call with a short farewell. Squid lets the farewell play out, then hangs up both the OpenAI and the Twilio legs.

An outbound call placed with placeCall follows the same path from step 1 on once the callee answers, with the call's purpose added to the model's instructions.

What the realtime model does and what the agent does​

The model is told to answer greetings, small talk and confirmations itself, and to call ask_agent for everything it cannot know on its own, passing the caller's request with every detail. That keeps the conversation fluid while the agent stays the single source of truth. Consequences worth knowing:

  • Put facts, rules and tools on the Squid agent, not in the phone line. The model never sees your knowledge bases directly; it asks.
  • The agent's instructions apply to what it answers; how the model speaks is governed by Squid's built-in phone instructions and the line's language settings.
  • A tool that takes long makes the model say "one moment" first; keep AI functions used on calls fast.

Browser calls​

The same connector serves calls from a web page, no Twilio and no phone number needed. The browser opens the microphone, Squid brokers a WebRTC session with OpenAI and runs the agent behind it, and the agent's voice plays in the page:

Client code
const call = await squid.ai().voice().startWebCall({ agentId: 'front-desk' });
call.onTurn((turn) => console.log(`${turn.role}: ${turn.text}`));
call.onEnd(() => console.log('The call ended'));

Users may start calls under the agent's security rules, exactly as they may chat with it; the agent then runs as that user. Every setting of a browser call, from the voice to the languages and the greeting, defaults to the agent's phone line when it has one with the OpenAI engine, and to this connector otherwise, and can be overridden per call. See Voice agents for the full API.

Using the OpenAI Voice connector from code​

npm install @squidcloud/openai-voice-client
Backend code
import { SquidOpenAiVoiceClient } from '@squidcloud/openai-voice-client';

const openAiVoice = new SquidOpenAiVoiceClient(this.squid, 'openai_voice'); // your connector id

// Where phone calls go, and the webhook URL to register in the OpenAI project.
const { sipUri, webhookUrl } = await openAiVoice.sipTarget();

// The realtime models the account can put on a call, newest first, and the voices.
const { models } = await openAiVoice.listModels();
const { voices } = await openAiVoice.listVoices();
MethodExecutable nameDescription
sipTarget()openAiVoiceSipTargetThe SIP URI calls are sent to and the webhook URL to register. Needs the API key.
listModels()openAiVoiceListModelsRealtime models of the account, newest first.
listVoices()openAiVoiceListVoicesVoices of the realtime models.

Phone lines are set up through the Twilio connector, and calls are answered by Squid itself; these methods are the building blocks the Twilio connector and the console use.

Troubleshooting​

SymptomCause and fix
The phone rings and is never answeredThe project has no webhook endpoint for realtime.call.incoming, or it points elsewhere. Register the URL from the ability's status line, see registering the webhook.
The application log shows INVALID_OPENAI_WEBHOOK_SIGNATUREThe connector's Webhook Secret is not the signing secret of the endpoint that sent the call. Each endpoint has its own secret; copy it again after creating or rotating the endpoint.
The log shows "Call … names no agent of this integration; rejecting it"The call's SIP headers name an agent whose phone line does not use this connector, for instance a line saved with another OpenAI Voice connector id, or a call from another application sharing the project.
The ability reports "Agent … is not wired up for OpenAI yet"The line was saved before the OpenAI connector was chosen. Save the ability again.
OPENAI_VOICE_INTEGRATION_REQUIRED on a browser callThe agent has no phone line with the OpenAI engine and the application has several OpenAI Voice connectors. Pass integrationId to startWebCall().
The model says it will check and then answers with an error or nothingask_agent failed: the agent's turn raised an error. Look at the application log for the agent's failure, such as a broken AI function or a model provider error.
The call cuts off a moment after the farewellExpected: the call is hung up once the farewell has played out (or after 15 seconds when OpenAI reports no playback).