Choose the voice architecture before writing the UI
The Realtime API is a good fit when you want speech, reasoning, and tool use inside one live session. It gives an application a streaming conversation where a person can speak, interrupt, and hear a spoken response without building a transcription request, a text model request, and a separate text-to-speech request for every turn. OpenAI's voice-agent guide also describes chained voice pipelines, where speech recognition, an agent, and speech generation remain separate steps. Choose a chained design if you need to inspect or transform the transcript between stages, swap each provider independently, or retain a text-first policy layer. Choose Realtime when direct speech-to-speech interaction is the simpler product requirement.
This distinction affects your cost ledger as well as latency. A chained design can produce a transcription charge, a model charge, and a voice charge. Realtime usage is attached to the responses generated during a session, with separate treatment for optional input transcription. The design should decide which provider usage records are authoritative before you build dashboards or customer-facing summaries.
Step 1: keep credentials and session policy on the server
Never put a standard OpenAI API key into JavaScript shipped to a browser or mobile app. The browser is an untrusted client: a user can inspect its code, requests, and storage. Make a small authenticated route in your application server that creates a Realtime session or mints a temporary client secret. The server authenticates to OpenAI with the standard key; it returns only the limited credential or session material the client needs. OpenAI's current WebRTC documentation describes both a unified session route and ephemeral client secrets. For a browser, it recommends WebRTC rather than WebSockets for more consistent performance.
Use this route as a policy boundary. Check the signed-in user, the right to start a call, any product-specific concurrency limit, and the allowed session configuration before you ask the provider for a credential. Keep the model, voice, instructions, and enabled tools on a server-controlled allowlist. Do not accept an arbitrary model name, tool list, or prompt from a public request and forward it unchanged. Apply a per-user or per-account start limit, protect the route from cross-site requests, and avoid writing ephemeral credentials to logs. If you need an OpenAI safety identifier, set it on the server request that creates the client secret; the provider's guide says the identifier is bound to the resulting temporary token.
Step 2: connect the browser with WebRTC
The browser creates an RTCPeerConnection, requests microphone access only after the person takes an explicit action, adds the local audio track, and creates a data channel for session commands and transcript events. It then exchanges the session description with your server or with the Realtime endpoint using the short-lived credential. The remote audio arrives as a media track. The data channel is for control messages and events, not for manually forwarding each audio frame when the WebRTC media path is active.
In practical terms, separate the connection lifecycle into visible states: permission requested, connecting, connected, reconnecting, closing, and closed. Provide a clear stop control and a visible microphone indicator. Handle a denied permission request, browser autoplay restrictions, an expired temporary token, network changes, a provider error, and a user who closes the tab. Treat the browser's local clock and its own estimate of call length as UI aids; they are not your billing record.
For a first version, keep the screen small: a start button, connection state, mute control, transcript area if your product needs one, and a clear end-call action. Do not store a transcript merely because the API can provide one. Decide whether it is necessary, who can see it, how long you retain it, and how a user can request deletion. Voice data and transcripts can contain personal information even when the product is framed as a simple support assistant.
Step 3: make tools narrow and permission-aware
A voice model may request an application function such as checking an order or looking up an appointment. The request is not authorization. Your server must validate the function name and arguments, confirm that the authenticated person can access the referenced record, and enforce business rules before it performs a read or write. Return only the fields the agent needs. For actions with an external consequence, such as cancelling an order, changing a booking, or issuing a refund, add confirmation or a human review step appropriate to the action.
Design a tool response for failure as carefully as for success. A timeout should not be narrated as a completed transaction. Return a short, typed result that distinguishes found, not found, unavailable, and requires confirmation. Set a deadline for each tool and decide whether a safe retry exists. If the same call could create a duplicate side effect, use an idempotency key in your own business service. The model can say that it is checking; your server decides whether a transaction actually happened.
Test interruptions and overlapping speech. When a person interrupts an answer, stop or cancel work that no longer matters, avoid speaking the cancelled answer over the new request, and keep your tool state tied to the correct conversation turn. Add a maximum session duration and an explicit inactivity policy. Neither setting replaces usage metering, but both make the product behavior understandable and help contain accidental long-running sessions.
Step 4: collect the provider's usage, not just elapsed minutes
Realtime bills conversational responses by input and output tokens. The conversation history is carried into later responses, so a later turn can include prior items as input; stable earlier content may be cached, while changing instructions or tools can reduce cache reuse. OpenAI's cost guide says user audio tokens are approximately one token per 100 milliseconds and assistant audio tokens one per 50 milliseconds, with small variations from special tokens. If input transcription is enabled, it is measured separately. A session lasting five minutes can therefore cost differently from another five-minute session because the model, speech mix, response length, conversation history, cache use, and transcription choice differ.
Listen for each completed response event and persist its usage alongside a provider session identifier. Capture input and output token categories, cached input where reported, the selected model, and whether separate transcription was enabled. Keep a durable application record keyed by the session and response IDs so a reconnect or repeated event does not silently add usage twice. Do not infer missing usage from audio duration and present it as an exact charge. Mark estimates as estimates and incomplete sessions as incomplete until a final provider event or provider log is available.
For a session total, sum the actual response usage records and any separately billed transcription records. Then apply the exact model's current input, cached-input, and output rates in your own rating or finance code. Store the rate version and currency used for that calculation. OpenAI changes its catalog, and model identifiers can be confused with similar names; a rate lookup should be explicit and versioned rather than guessed from the word “Realtime.” A cost estimate made before a call is useful for product planning, but provider-reported usage after the call is the better reconciliation input.
Step 5: meter quantities without pretending a meter is a price
Keep two layers distinct. Your application or finance system can calculate a provider cost from the token usage and rate card. A usage meter records a chosen quantity, such as calls completed, input audio tokens, or output audio tokens. These are useful for analyzing volume and building a separate commercial model, but a token count is not itself a dollar charge. UsageBox aggregates submitted usage quantities; it does not apply the OpenAI price card, calculate your margin, or produce an invoice.
With UsageBox, create numeric meters for the quantities you want to roll up, then submit a final session summary from your trusted server. The endpoint accepts an array of records; each record uses an account's own meter ingest key and a numeric value. Keep your UsageBox API key in the server environment. A representative batch can look like this after your application has verified the final provider usage:
[
{ "meter": "YOUR_SESSION_METER_INGEST_KEY", "value": 1 },
{ "meter": "YOUR_INPUT_AUDIO_TOKEN_METER_INGEST_KEY", "value": 8420 },
{ "meter": "YOUR_OUTPUT_AUDIO_TOKEN_METER_INGEST_KEY", "value": 6150 }
]Send the batch with the server-side x-api-key header and a stable Idempotency-Key for that completed session summary. Use real ingest keys from the account that owns the resources; never copy an example key from documentation. If the application has several candidate meters, pass the intended ingest key explicitly. A successful HTTP status alone is not enough for your own accounting: inspect the response counts and skipped details so the caller knows whether the events were processed.
The usage event schema is intentionally small: routing keys, one value, and an optional timestamp. It does not accept arbitrary model names, customer labels, session IDs, or dollar amounts as extra event dimensions. Keep the session-to-customer mapping and full token breakdown in your own application database, and choose the supported resource keys for the aggregates you actually need. That boundary prevents a common failure: sending a large JSON object, getting an accepted response, and assuming fields the meter does not ingest were retained.
Build a reconciliation loop before charging customers
For the first release, compare three views for a sample of calls: the response usage events, the session summary in your application database, and the aggregated meter quantities. The sums should agree for the dimensions you meter. If the token events do not add up, investigate missing final responses, duplicate delivery, transcription counted twice, or a retry that used a new idempotency key. Keep provider cost reconciliation separate from customer pricing so you can change a plan without rewriting the original usage record.
Sample across short and long calls, quiet pauses, interruptions, tool use, cache hits, rejected microphone access, and sessions that end unexpectedly. Compare cost estimates with final API usage. Include a test where the browser disconnects while the server still has a provider session, and define which system closes it. If no final usage arrives, label the record as pending or estimated rather than silently treating it as zero.
When the application is ready for commercial use, make the product rule explicit: which quantity is metered, which component determines a customer's price, when usage becomes billable, and how a user can dispute a record. UsageBox can provide quantity aggregation for those chosen meters. Your product's own billing layer remains responsible for converting the quantities into a price and communicating that price to the customer.
Launch checklist
- Keep standard provider credentials on the backend and issue only limited session credentials to the client.
- Allowlist models, voices, prompts, and tools; authenticate each session start.
- Use WebRTC for browser media and a data channel for supported control and event messages.
- Close sessions gracefully and retain provider-reported usage for each completed response.
- Record token quantities and provider costs in separate fields and separate layers.
- Send an idempotent final usage summary from a trusted server and read skipped-record details.
- Do not send arbitrary metadata and assume UsageBox will retain it; the usage schema is deliberately limited.
- Reconcile sample sessions against provider logs before using the data in customer-facing billing.
Realtime is the transport and model layer. Your server owns permissions, tool effects, and the durable session record. Your pricing system owns rates. A meter owns a quantity. Keeping those jobs separate makes both the voice experience and the resulting usage totals easier to audit.
For the official browser connection flow and current session mechanics, read OpenAI's Realtime WebRTC guide. For current provider rates and a like-for-like voice stack comparison, see the 2026 voice-agent pricing guide.