Realtime voice, under 200ms

VoiceOps AI

A real-time voice-streaming backend holding sub-200ms latency across 100+ concurrent sessions — WebSocket ingest, Kafka consumer groups, and Kubernetes HPA that autoscales on consumer lag.

A live voice stream never pauses — audio keeps arriving whether the backend is ready or not. VoiceOps had to run streaming inference on that audio and return results inside a conversational budget (sub-200ms), and hold it not for one stream but for 100+ at once, where any backpressure at the source is a dropped call, not a queued job.

end-to-end latency: <200ms; per session
concurrent sessions: 100+
autoscaling: consumer-lag; Kubernetes HPA
transport: WebSockets; Kafka fanout

the bar it had to clear

Sub-200ms end-to-end per session — conversational latency is the product, not a nice-to-have.
Input never pauses: audio frames keep arriving whether the pipeline is ready or not — it absorbs or sheds, never blocks the source.
Hold that budget across 100+ concurrent sessions — sized for bursts, not averages.
Observability is first-class: consumer lag and P95 latency are tracked across the whole distributed pipeline.

how it actually works

hover or tab a step to see how it works — or trace a route above.

01
Ingress that never waits
Live audio streams in over WebSockets, one connection per session. Backpressure here is a dropped call — the pipeline absorbs or sheds, it never blocks the source.
02
Decouple ingest from inference
Each session’s frames are published to Kafka; consumer groups fan the work out to a pool of inference workers, so a slow worker creates lag — not a dropped call.
03
Output before input ends
Inference runs in a streaming fashion — results begin flowing before a session’s audio ends. That overlap is where the sub-200ms budget is won.
04
The sub-200ms promise<200ms
Results stream back per session at sub-200ms end-to-end, 100+ sessions at a time.

watch it listen

Realtime speech opens a voice-activity gate, then streams. Hit speak — it detects the voice and transcribes token-by-token.

idle

Illustrative — the transcript streams token-by-token, the way the realtime pipeline works. The real budget lives in the numbers above: sub-200ms end-to-end across 100+ concurrent sessions.

the problem

Live audio is relentless — a voice stream never stops to let the backend catch up. VoiceOps had to run streaming inference on that audio and deliver results at conversational latency across 100+ simultaneous sessions, where a naive synchronous pipeline either falls behind under load or drops frames at the source.

the approach

Audio streams in over WebSockets; each session’s frames are published to Kafka, whose consumer groups fan the work out to a pool of inference workers.
Inference runs in a streaming fashion — results begin flowing before a session’s audio ends, which is where the latency budget is won.
Kubernetes HPA autoscales the worker pool on Kafka consumer lag — the most honest backlog signal — so capacity tracks real demand, not CPU guesses.
State and results land in PostgreSQL, MongoDB, and Redis; consumer lag and P95 latency are tracked across the pipeline so regressions surface before users feel them.

the calls that mattered

Every architecture is a set of trade-offs taken on purpose. These are the load-bearing ones.

Fan out over Kafka consumer groups, not a synchronous call.

A voice stream can’t wait on a busy worker. Publishing frames to Kafka and consuming them in groups decouples ingest from inference, so a slow worker creates lag — a number you can scale on — instead of a dropped call.

the trade-offA broker in the hot path and delivery semantics to reason about — accepted because it turns backpressure into an observable, scalable signal.

Autoscale on consumer lag, not CPU/GPU utilization.

Utilization lies under bursty streaming load; Kafka consumer-group lag measures work actually waiting — which is what latency is made of. Kubernetes HPA scales the worker pool on lag.

the trade-offNeeds lag-based HPA tuning and a broker to watch — more moving parts, for autoscaling that tracks the latency users actually feel.

Stream results — emit before the session ends.

Waiting for a full utterance blows the sub-200ms budget by construction. Streaming inference overlaps “still listening” with “already responding” — the overlap IS the latency win.

the trade-offDownstream consumers must handle incremental, revisable output instead of one final blob — a harder contract for a far lower latency.

what shipped

VoiceOps holds sub-200ms latency across 100+ concurrent voice sessions — WebSocket ingest, Kafka consumer groups, and Kubernetes HPA autoscaling on consumer lag, backed by PostgreSQL, MongoDB, and Redis. Consumer lag and P95 latency are tracked across the distributed pipeline, so the tail budget holds through traffic bursts.

what’s next

Speaker diarization on the stream so multi-party sessions are attributed per-voice.
Adaptive batching under load — trade a few ms of latency for throughput only when lag rises.
Per-session backpressure policies so a slow consumer degrades gracefully instead of dropping.

built with

KafkaWebSocketsKubernetesFastAPIPostgreSQLMongoDBRedis