Build realtime voice agents on AI Gateway
AI Gateway now supports audio/voice. You can add realtime voice, text to speech, and speech to text with the same calls you already use for text, image, and video, routed through AI Gateway alongside every other modality.
Audio launches with models from OpenAI and xAI. Each call gets the same provider routing, observability, spend controls, and bring-your-own-key support you already use for your other models.
These capabilities are in beta and available in AI SDK 7.
Capability
How it works
Use it for
Realtime voice
Live audio in and out, for streaming, low-latency session
Two-way voice agents and live conversation
Text to speech
Text in, audio file out, single request
Voiceovers, spoken responses, audio versions of written content
Speech to text
Recorded audio in, text out, single request
Transcribing voice notes, call recordings
Getting started
Realtime, speech, and transcription model are supported on AI SDK 7.
npm install ai @ai-sdk/react @ai-sdk/gatewayRealtime voice agents
Realtime turns your app into something a user can hold a conversation with. When they speak, the model responds right away. Because it replies in the moment instead of waiting for a full turn, users can interrupt and talk over it the way they would with a person. It fits voice assistants, customer support agents, hands-free tools, and anywhere a user would rather talk than type.
What sets it apart from chaining models together is that a single realtime model hears audio and produces audio directly, instead of running a speech-to-text, then language model, then text-to-speech pipeline.
In the browser, the useRealtime hook manages the WebSocket connection, microphone capture, and audio playback.
The connection is authenticated with your AI Gateway credential, so you mint a short-lived token on the server and hand the browser only that token. Your API key never reaches the client. Add a route that mints the token:
1import { gateway } from '@ai-sdk/gateway';2
3export async function POST() {4 const { token, url } = await gateway.experimental_realtime.getToken({5 model: 'openai/gpt-realtime-2',6 });7 return Response.json({ token, url, tools: [] });8}Then connect from a client component: