Voice

Preview — enabled per account

Voice is not generally available. The Voice switch does not appear under Capabilities until your account is added to the rollout, and until then a token cannot carry the voice capability at any ceiling — so there is nothing to turn on and nothing to misconfigure. Ask us to enable it and the switch appears where this page says it is.

The reason it is gated rather than simply off by default: a call holds a live speech session for its whole duration, so it is the one capability whose cost is a machine rather than a request.

With the voice capability your users can talk to the agent. Not speech-to-text, a model, and text-to-speech glued together — a full-duplex speech model that hears them while it is speaking, so they can interrupt it mid-sentence and it stops.

The part that makes it usable for real work is the split:

	Owns
The speech frontend	The talking. Timing, interruptions, backchannels, prosody, the "yeah, let me check" while it waits.
The model you chose	The answer. Whatever model your project is configured for, with your documents, tools, memory and sandbox behind it.
Oberik	The waiting. Bridging a 300ms gap, holding for a 30-second lookup, transferring to a person.

So the voice does not change when the substance arrives. The caller hears one agent saying "yeah, let me check — it left the warehouse yesterday and it's expected Friday", and the second half of that sentence came from your model. Change the model and the voice is identical.

Enable it

Capabilities: turn on Voice for the project. Turn on Transfer to a person too if the agent should be able to escalate.
Token: include voice (and voice:transfer) in the capabilities you mint.
Open a call: POST /voice/sessions, then stream audio.

const call = await af.voice.open({
  voice: "warm",
  role: "support",
  waiting: { strategy: "auto", barge_in: true },
});

call.on("state", (s) => setStatus(s.status));       // "Listening" · "Thinking" · "On hold"
call.on("caller.transcript", (t) => addLine("them", t.text));
await call.send(micPcm);                            // 16-bit PCM, any rate

The token is the same end-user JWT everything else uses, and the turn behind the voice is an ordinary turn — same capability gate, same tool allow-list, same per-customer cost attribution. There is no separate voice agent to configure.

Two options worth knowing about at setup:

const call = await af.voice.open({
  // Narrow what this call may use. Intersected with what the token already allows, so a call
  // can decline a tool and never buy one. Omit for everything the token allows.
  allowed_tools: ["search_documents", "get_order_status"],
  // Which checkpoint on the speech host, where it serves more than one. `af.voice.frontend()`
  // lists them. Asking for one it does not serve is a 400 rather than a silent fallback.
  voice_model: "support-tuned",
});

Narrowing is worth doing. A support line has no business running sandboxed compute, and every tool the model can see is a tool it can spend a caller's silence on.

Which speech model

The Speech frontend setting on a project picks which one runs its calls. Blank uses the deployment's default, and every project should be on that unless it is deliberately comparing.

Frontend	What it is	How a lookup starts
`personaplex`	A self-hosted full-duplex model. The answer is spoken into it word by word as it streams.	The runtime decides, from the caller's words and the model's own.
`openai`	OpenAI's hosted realtime API.	The model asks, by calling a tool; the runtime stands down.
`nemotron`	NVIDIA's NemotronLabs VoiceChat container.	The model asks, as above.

The difference in that last column is the one that matters. On a frontend that asks for itself, the model owns the decision: a social turn delegates nothing, and a turn it does not ask about is not looked up. On personaplex the runtime routes, because that model has no tool channel.

openai needs a key, and if you have already registered one it is used. A project with an OpenAI provider under LLM & limits needs nothing here: calls use that credential. The box on this tab is for the case where you want a different key for calls than for the rest of the project — it is billed to your own account and is never sent back to the dashboard, so an empty box means "keep the stored one". So the order is:

a key set on the Voice tab, when you deliberately set one;
otherwise the project's registered openai provider key;
otherwise the deployment's own, if it has one.

A project with none of the three is refused at POST /voice/sessions with a 400 rather than failing mid-call. GET /voice reports which of the three answers, as openaiApiKeyFrom.

Whichever you pick, Oberik is reached the same way: the model calls one tool with the caller's intent in plain words, or a second tool that takes no arguments at all and lets Oberik read the conversation. The answer comes back as a fact to relay or as a question to put to the caller, and the model does the relaying in its own voice.

Saying something the agent could not know

Two channels, deliberately different. inject() is "know this"; say() is "say this". Conflating them is what makes an agent recite a reference number.

await call.inject({ tier: "enterprise", customer_id: "c_8812" });  // shapes tone, never spoken

const r = await call.say("Sorry to interrupt — that card was just declined.");
if (!r.spoken) console.warn("not said:", r.reason);

say() is for what only your application knows in the moment: a card that was just declined, an appointment that has moved, a line compliance requires on every call. Not for answers — an answer is a turn, and it gets a model, a bridge while it is being composed, and a latency you can measure. Text pushed through say() gets none of those.

It goes through the same machinery the agent's own speech does, so the caller can talk over it. That is the point: a line the caller cannot interrupt is a line they keep hearing while they are trying to object to it.

By default it takes the floor — an answer being spoken is abandoned, and displaced_chars says how much of it was. That is right for what this is for; a declined card outranks the sentence it lands in the middle of. Pass { interrupt: false } for a disclosure that should wait, and it is refused rather than chopping an answer in half.

Always check spoken. On hold, mid-transfer, or against a speech host that cannot be forced, the honest answer is no and reason says which — and an application told its compliance line was read when it was not has no way to find out.

What a caller actually hears

A call where the answer takes a second and a half:

0ms      the caller stops talking
~200ms   "Yeah, absolutely—"                 ← the frontend, covering the gap
~250ms   the model starts, calls a tool
~1500ms  "...it left the warehouse yesterday and it's expected Friday."
                                             ← the model's words, the same voice

They never experienced a 1.5-second delay. And if they interrupt halfway through — "actually, never mind, I found the email" — the agent stops inside 200ms, the answer is abandoned, and nothing from it is ever spoken. Not "queued and skipped": a superseded answer cannot reach the caller, by construction.

Waiting

A gap of 300ms and a gap of 30 seconds are different problems, and covering both with a longer "let me see" gets the second one wrong. Oberik picks from four behaviours, using what it has actually measured for your models and tools rather than a guess:


Bridge	Under two seconds. The agent says something noncommittal — “yeah, let me check” — and the answer arrives inside it, so the caller never hears a gap
Wait	Two to ten seconds. The agent says once that it is looking something up, then goes quiet
Hold	Longer than ten seconds. The agent tells the caller it is putting them on hold, hold media takes over the line, and the work carries on independently
Transfer	A hold whose resolution is a person rather than a result

The filler is deliberately noncommittal. "Yeah—" and "okay, so—" leave the sentence open for whatever the model says next; the agent never guesses at the answer, because whatever it guessed the model might contradict and the caller would have heard both.

Hold

A hold is a state, not a sentence. The caller hears hold media, the work carries on, and — this is the part most systems get wrong — they can still speak. A caller who says "never mind" into thirty seconds of hold music and is not heard has been hung up on by a system that thinks it is being helpful.

Four listening policies, because the cost is real: keeping the full speech session live for a ninety-second hold is a machine doing nothing.

`listen`
`full`	The speech model stays live. Barge-in, corrections, everything
`interrupt_only`	Cheap detection for a wake or a cancel, rather than the full model. The default
`dtmf_only`	Keypad only — "press 1 to come back"
`off`	Nothing. Only where being interrupted would break something

Coming back is one moment, not two: the media fades and the agent speaks into it. "Thanks for waiting — I found it, the replacement is with the courier."

When the slow thing is yours — a CRM lookup, a booking system, your own backend — let the SDK own the race:

const account = await call.holdUntil(
  () => crm.expensiveLookup(customerId),
  { reason: "looking up the account", maxSeconds: 90 },
);
if (account.outcome === "done") {
  // The caller is already back and has heard "thanks for waiting". Say the answer.
}

Four things can happen and all four come back rather than throwing: the work finishes, it fails, the ceiling is reached, or the caller talks their way out of it. That last one is why this exists — getting it right by hand means racing four conditions and cancelling in the right order.

Your agent can ask for a hold itself when it knows something is slow:

Tool
`voice_hold`	put the caller on hold because this will take a while
`voice_resume`	come back, then say the answer
`voice_transfer`	hand the call to a person (needs `voice:transfer`)

It does not need these for an ordinary tool call — short waits are covered automatically. They are for when the agent already knows the alternative is silence.

Deep hold

A ninety-second wait for a human agent is ninety seconds of an expensive speech session generating nothing, and the hold media does not come from the model — so there is nothing for it to do. A deep hold releases the session and restores it on the way back.

Two conditions, and Oberik refuses rather than pretending when either fails:

The frontend must genuinely be able to restore. Parking one that cannot come back means a resumed call arrives as a subtly different agent mid-conversation, which is worse than an expensive one.
The hold cannot be listening. A parked session has nothing to hear with, and a hold that claimed to keep listening would be deaf and say otherwise.

Where either fails the session stays resident and the call reports why. So a deep hold is worth asking for on a listen: "off" or dtmf_only wait for a person, and is not available on the short lookups where you would want barge-in anyway.

Transfer

A warm transfer: the caller waits on hold while the agent briefs whoever is picking up with the context you asked for, then they are connected. If nobody takes it, the caller comes back to the agent rather than to a dead line — which is the whole reason to do this here rather than in your phone system.

What the agent says itself, and in which language

Almost everything on the call comes from your model. A handful of things do not — the filler that covers a gap, the sentence announcing a hold, the line it comes back on, the apology when the model falls over. Those are Oberik's own words, and there are two reasons you would want to change them:

Language. An English "Yeah—" spoken into a Turkish call is not a small blemish, it is the feature working incorrectly.
Register. "Yeah—" is wrong for a private bank, and every other utterance on the call already follows your system prompt.

So they are a project setting, editable in the dashboard under Voice — one box per line, with what each one is for beside it, and a badge saying whether it is yours, the built-in, or a line the agent currently stays silent on. Or over the API:

curl -s localhost:8000/api/projects/$ID/voice -X PUT \
  -H "X-API-Key: $KEY" -H 'content-type: application/json' \
  -d '{
        "language": "tr",
        "phrases": {
          "bridge.open": ["Tabii—", "Şey—"],
          "hold": "Bu biraz uzun sürecek. Kontrol ederken sizi beklemeye alıyorum.",
          "resume.done": ["Beklediğiniz için teşekkürler—"]
        }
      }'

voice show lists every line with what it is for. language blank takes the speech host's answer, which is the right default — a voice is trained in a language and the host is the one that knows.

Two rules, and they are enforced. A bridge.* or resume.done line must end open — with a dash, an ellipsis or a comma — because the model's answer continues the same sentence. And a bridge.* line must commit to nothing: it is spoken before the answer exists, so if it says "yes" and the model then says no, the caller has heard both. "I think it's probably—" is refused for the same reason, and so are "Unfortunately—", "Good news—" and "That order has shipped—".

Both are checked when you save, not on the call — so you find out at the box you typed it in.

Which of the two ran, in your language

The shape rule is typographic and holds for any language. The commit-to-nothing rule is English morphology, and it only runs for English wording: it can see that "Yes, it's ready—" answers the question and it cannot see that "Evet, hazır—" does.

So a project configuring non-English phrases gets a warning on the save saying exactly that, rather than a silent pass that reads like approval. The rule still applies — a filler that answers the question before the model has is still wrong, and still worth writing carefully — it is just not something we can catch for you. voice show prints what each line is for, which is the thing to write against.

What happens in a language we have no words for. Nothing is said. Not English — a sentence the caller cannot understand is worse than silence, so the agent stays quiet where it would have spoken and the call reports which lines those were, in phrases_missing on the session and as an error event. A missing filler means the speech model covers its own gap, which it is good at; a missing hold announcement means the media takes over unannounced, which is worse and still better than a sentence nobody follows.

The built-in set is English only. Configuring your own is how any other language works, and the dashboard's Voice tab — or voice show — tells you exactly which lines are still silent.

Give a line several entries and it rotates, so a caller who waits twice does not hear the same sentence twice. In the dashboard that is one per line in the box; over the API it is a list.

Numbers said correctly

A speech layer must not round a balance. Mark anything that has to survive exactly and it is spoken verbatim, never paraphrased, and never split across a pause:

Your balance is <verbatim>£1,847.23</verbatim> and the invoice is due
<verbatim>31 August</verbatim>.

If a caller interrupts partway through one of those, the call reports that the figure was not conveyed — rather than assuming that no error means it was heard. That is checked against what the frontend actually said, frame by frame, not against what was queued: the two differ by exactly the interesting amount whenever somebody interrupts.

Three fidelity modes: strict (every character, for figures and disclosures), natural (the default — punctuation may adapt, meaning may not), and free (the agent may paraphrase). In strict, a figure that did not reach the caller produces an error event on the call, because that is precisely the thing you asked to be told about.

On the telephone


Browser / app	WebRTC or a WebSocket, full quality
Phone number	Twilio Media Streams, or RTP behind your own SIP termination

Oberik carries the audio of a SIP call — RTP, µ-law, keypad input as RFC 4733 events. It does not do the signalling: no INVITE, no SDP, no registration. Point whatever already terminates SIP for you at it — an SBC, a media gateway, FreeSWITCH, Twilio's SIP interface — which is the same division Twilio itself makes between Media Streams and TwiML.

One consequence worth knowing before you build against it: on RTP the agent cannot be told when audio has played, only that it was sent. So verbatim figures are still checked against what the frontend said, and never against a playout report that does not exist.

A PSTN call is 8 kHz µ-law — a quarter of the bandwidth the speech model generates. That is the telephone network, not a setting, so a phone call will not sound like the browser demo. What Oberik does guarantee is that it sounds like a phone call rather than like aliasing, and every call reports whether it is on a narrowband channel.

Barge-in on a phone needs one thing most integrations miss: when the caller starts talking, audio already sent to the carrier has to be dropped, not just stopped at the source. Oberik sends Twilio's clear for exactly this. Without it a caller interrupts and then listens to a second of the agent still talking, which reads as rudeness rather than as a limitation.

Watching a call

Every call emits a stream of events — the same ones the dashboard's Try agent page renders. The ones worth wiring into your own UI:

Event
`state`	the call moved: `listening` · `delegating` · `bridging` · `external_speech` · `holding` · `resuming` · `transferred`
`caller.transcript`	what the caller said
`voice.bridge`	the filler it chose, and how long it expects to need it
`voice.takeover`	the model's words are now being spoken
`voice.interrupted`	the caller barged in; what was spoken and what was dropped
`hold.entered` / `resume.done`	hold media on and off
`delegate.failed`	the model could not answer, and the agent is recovering conversationally

And a set of latencies, because a voice agent lives or dies on them and an average that excludes the turns it dropped gets better as the system gets worse:


`response_start`	caller stops → the agent says anything at all
`answer_start`	caller stops → the agent says something with substance in it
`barge_in_stop`	caller starts → the agent has stopped
`tool_to_spoken`	a tool finished → the answer is being spoken. Separates a slow tool from a model that is slow to compose from a result it already has
`dropped_turns`	requests that never produced an answer. Reported beside the timings, never folded into them
`unwanted_barge_ins`	the agent stopped for something that turned out not to be a request — an acknowledgement, a "mm-hmm". Counted apart from `barge_ins`, because that raw count going up can mean the feature is working and this one going up cannot
`false_interruptions`	the agent stopped and the caller then said nothing at all. The number that says whether the detection threshold is set for the room the call is actually in

And one number that is not a latency. word_error compares what the model produced against what the caller actually heard, per turn:

Because the frontend is handed the model's own characters, this is designed to be zero. Anything above it is a lossy path — a dropped chunk, a cut span — and not a paraphrase, which is exactly why it is worth reporting. Turns the caller interrupted are reported separately and excluded from the call's figure: an answer abandoned on purpose is not a transcription failure.

Sent, said, heard

Three different facts, reported separately, because a system that cannot tell them apart will act on the first when only the third matters.

sent — the runtime released the text.
said — the agent realised it as audio. verbatim figures report delivered.
heard — the caller's own player got that far. verbatim figures report heard.

The third needs evidence from your side of the wire. The server places a mark at the point each protected figure finishes being spoken and tells you about it (media.mark_placed); confirm it with markPlayed(mark) once your player has played past it. The SDK's browser player does this for you.

heard is three-valued and null means unknown — either your transport cannot report playout, or the confirmation has not arrived yet. Not false: a network round trip is not a defect, and rendering it as one shows every figure as lost for a moment after it was spoken. Once the call has ended, an outstanding mark is a false — the confirmation is not coming.

If you cannot honestly tell when audio played, do not confirm. A mark confirmed early tells the server a sentence was heard while it was still queued, which is the one fact a compliance readback must not get wrong.

Limits

One live speech session per call, for the call's whole duration. This is the one capability whose cost is a machine rather than a request, so a deployment caps how many it accepts — voice_max_concurrent.
A call has a duration ceiling. A token can narrow it (max_voice_seconds) and never widen it.
The agent does not work through bot checks, on a call or anywhere else.

Enable it​

Which speech model​

Saying something the agent could not know​

What a caller actually hears​

Waiting​

Hold​

Deep hold​

Transfer​

What the agent says itself, and in which language​

Numbers said correctly​

On the telephone​

Watching a call​

Sent, said, heard​

Limits​