Beyond audio in, text out.

These are real production transcripts — single detached utterances from different calls, with anything identifying removed. They happen to come from one booking-line deployment; every domain we serve has its own versions. When we replayed the same audio through a model we control — telling it, turn by turn, what the call was about — all three came back correct. More on that in a minute.

In a cascading voice-AI flow, when your transcription is wrong, everything downstream is wrong. The LLM reasons about the wrong words, the agent acts on them, and the caller repeats themselves until they hang up. And with a hosted API there is nothing you can do about it.

Open weights changed that math. Qwen3-ASR is an Apache-2.0 speech recognition model with strong English, Mandarin, and Cantonese; the 1.7B-parameter version fits comfortably on a single mid-range GPU — we serve it on an NVIDIA L4. We already orchestrate many STT engines and pick the best one per deployment — we’re not trying to win the model race. But for this slice of traffic, none of the engines we route to fit. So we hosted one ourselves — and today, every call that used to go to the hosted API runs on it.

A production speech stack has four jobs. Be accurate — hear the caller right, in whichever language they’re speaking. Be fast — the transcript has to land before the pause turns awkward. Be concurrent — both of the above, for every simultaneous call, at a cost that doesn’t meter the minutes. And handle turn-taking — know when the caller is actually done speaking. This post is how we went after the first three; the fourth is where we’re headed, at the end.

We didn’t move production traffic on vibes. Before the first call switched over, the self-hosted model had to beat or tie the hosted API on a controlled benchmark: a public multilingual test set (FLEURS), 400 clips per language, every clip pushed through a simulated telephone channel to match real call audio — and scored so that a win has to survive statistical resampling before we’re allowed to call it a win. The result:

One detail we’re proud of: every time we found a flaw in our own benchmark harness, fixing it improved the hosted API’s numbers. One language flipped from “we win” to “tie.” We published the tie. A benchmark you want to win needs safeguards that stop you from accidentally winning.

Parity on benchmarks, better on our real calls. But parity was never the point. The point was the three things we could suddenly do.

Does it actually help, or does the model just parrot whatever you feed it? We ran it distractor-controlled: the same hard clips with the right context, with an unrelated context, and with none.

The context has to be right to help; a wrong one buys you nothing.

Then we replayed real production calls. The party-size error fixed itself every time the right vocabulary was in context — and the unrelated context did not fix it. That’s the difference between “the model got lucky” and “the context caused the fix.”

Why does such a simple design work? Because of an asymmetry in the workload. Continuous batching — requests joining and leaving mid-flight — earns its complexity when requests have wildly different lengths, like a chatbot writing one 50-token reply and one 4,000-token essay. ASR turns are short and similar-sized: a batch finishes together, so a fixed batch wastes almost nothing. We got nearly all of the win with a fraction of the machinery, and the streaming state — per-turn context included — never had to change.

So how does the finished streaming endpoint stack up against the hosted realtime API — qwen3-asr-flash-realtime, the hosted sibling of the same model family — measured from the same test client in the same run? First, latency: the finalization tail, end of speech → final transcript, with the audio paced in real time. Ours runs on a single NVIDIA L4.

Low, and roughly flat with turn length — by the time the caller stops talking, the growing window has already decoded nearly everything, so the final flush is close to free.

Part of that gap is simple geography. The self-hosted recognizer sits next to the workload it serves — same cluster as the agents — so a turn never pays a network hop out to a vendor and back. Co-location isn’t cheating; it’s the point: a hosted API is always on the other side of the internet, and yours doesn’t have to be. The gap widens with geography — not every STT provider serves every region, and when the nearest endpoint is an ocean away, every turn pays an intercontinental round trip. We once benchmarked a sub-second API from the wrong continent and watched it look like a three-second one. A model you host deploys wherever your callers are.

Then concurrency — per-stream tail as simultaneous streams pile onto that one GPU:

The before column is the cliff; the after column is flat. One L4 behind a queue, a window, and a worker thread holds its own against managed continuous batching, stream for stream. We then pushed one pod far past reasonable: 512 simultaneous streams. The pod didn’t fall over at any load we tested; it just gets slower. And how much slower is a tuning knob, not a ceiling.

Capacity planning becomes a latency budget, not a fear of falling over. Beyond one GPU’s budget, you add pods.

The economics land the same way. An NVIDIA L4 runs roughly $500–700 a month on-demand from a major cloud — less with commitments — and one of them just held 512 simultaneous streams. A hosted API meters every minute of every call; a GPU you own costs the same whether it’s carrying ten calls or two hundred. At voice-agent volumes, that line crosses fast.

At those positions, and only those, we mask the logits: the scores the model assigns every candidate next token. The mechanics matter. The mask doesn’t force a language; it narrows the menu. Allowed names keep their original scores and everything else drops to negative infinity, so the model still picks the language it genuinely thinks it heard — from choices that are all sane. Some names are one token and Cantonese is two, so the mask walks each allowed name token by token; the moment the boundary token appears, it switches off entirely and the transcript decodes untouched.

And the allowed list is not baked into the server — it’s a new parameter on the transcription request itself. The client code running the conversation sends it the same way it sends the per-turn context line: allowed_languages=yue,zh,en for this deployment. The server compiles those into token masks for that one request; requests without the parameter decode completely unconstrained. To a caller of the API, the whole guardrail is one more field. That field mostly has no equivalent elsewhere: common STT APIs offer a binary choice — pin one language, or auto-detect across everything the model knows. Where candidate-language lists exist at all, they’re connection-time settings again — not a per-turn, decode-level guarantee.

Getting this right taught us a lesson about constrained decoding. Our first attempt was a deny-list: push down the score of every unwanted language. It leaked. Suppress everything except Thai as a stress test, and the model escaped to “Cape Verdean” — a language name that wasn’t on our list at all. A deny-list can’t enumerate what it hasn’t imagined. The allow-list version — mask everything except the permitted names — is airtight by construction: in the same stress test it yields exactly what it’s told.

Two properties made this work. First, the language tag and the transcript are genuinely separable: force the language ID to English on a Cantonese clip and the transcript comes back character-for-character unchanged — you steer the decision without touching the words. Second, it’s free: measured with constrained and unconstrained requests interleaved in the same GPU batches, the overhead is within noise — a few percent either way, tails essentially identical.

We built this belt-and-suspenders: a recovery pass that catches a drifted turn after the fact and re-decodes it with the language forced (~0.3 s, on a path that rarely fires), and the logit mask that stops the drift before it happens — built, validated, and measured on the same serving stack. Neither is possible when the decoder is on someone else’s server.

In closingThe bottom line

It isn’t on someone else’s server anymore. We have since migrated all of the traffic that used to run on qwen3-asr-flash-realtime onto the ASR we host ourselves.

None of these three ideas is exotic. Per-turn context is prompt engineering for a recognizer. The batching engine is a queue, a window, and a worker thread. The guardrail is a mask over a handful of logits. What they have in common is where they live: inside the decode loop. And between them they cover the first three jobs — context and the guardrail buy accuracy; the batching engine buys speed and concurrency. The fourth is on the bench.