AI inference is moving beyond requests.
A growing class of models does not receive an input, produce an answer, and stop. It remains active alongside an external process: a conversation, a video stream, a robot, or an interactive environment. New input arrives incrementally, model state persists, and each output is useful only while it is still current.
We call this continuous inference.
Continuous workloads do not all have the same timing contract. A robot may have a hard control deadline. A video system may be allowed to drop a frame but not to fall several seconds behind. An interactive model may run on events rather than a perfectly regular cadence.
What they share is that throughput over a collection of completed requests no longer describes the job. The system has to keep a live session synchronized with something outside the GPU.
Full-duplex voice is the clearest current example because its contract is both strict and easy to hear. The model listens while it speaks, receives audio throughout the conversation, and produces output on a fixed cadence. A missed computation becomes a gap, a delayed response, or a model state that no longer matches what the caller heard.
For Moshi, the codec runs at 12.5 frames per second. Every 80 milliseconds, each active conversation supplies another frame and needs another model step. The next frame arrives whether or not the engine is ready for it.
Today, executing that step efficiently is already a major problem. A duplex path may contain an audio encoder, a large backbone, one or more dependent decoders, a codec, and session-state updates. The composition differs by model, so current implementations are largely custom-built.
General-purpose engines are optimized for throughput across batches of bounded requests, while a duplex runtime has to make this heterogeneous path fast at relatively small batch sizes and repeat it under a deadline.
Scheduling does not replace that execution work. A clock-aware scheduler cannot keep a deadline if the underlying step is too slow. Conversely, a fast model loop is not a serving system if it cannot decide how many sessions it can safely admit.
The unit of work has changed from a bounded request to a persistent session. That change affects execution, scheduling, memory, failure handling, and measurement. Duplex makes the break visible.
From requests to sessions
A conventional LLM request has a clear lifecycle:
- Prefill: process the prompt and populate the KV cache.
- Decode: generate tokens one at a time until the model stops.
- Release: free the request's resources or return its cache blocks to a reusable pool.
This lifecycle gives the scheduler room to trade a little latency for efficiency. Within its latency targets, it can wait briefly to form a larger batch, reorder queued work, preempt one request and resume it later, or send prefill and decode to different GPU pools.
Text serving still has latency objectives, sometimes strict ones. But a small delay is charged to the latency of that response; it does not create a missing output for an external time slot. The degradation is gradual.
A full-duplex session is different:
| Property | Conventional LLM request | Full-duplex session |
|---|---|---|
| Input | Prompt arrives complete | Audio keeps arriving |
| End condition | EOS, length limit, or cancellation | Caller hangs up |
| State lifetime | Request-scoped | Session-scoped |
| Scheduling trigger | Available queued work | Recurring frame clock |
| Timing objective | Response latency and token latency | Deadline on every frame |
| Overload behavior | Queue grows; responses slow down | Frames miss their playout time |
| Capacity unit | Requests or tokens per second | Concurrent sessions kept on schedule |
For Moshi, one frame is due every 80 ms. A 90 ms step exceeds that interval by 10 ms. That overrun consumes available playback-buffer margin; once the margin is exhausted, output arrives late. Repeated overruns accumulate unless later steps are fast enough to recover the lost time.
Figure 01 · The scheduling unit
A request ends. A session recurs.
Both workloads can batch work. A live session also has to keep pace with an external clock.
Request-driven
A request runs toward completion
The scheduler can wait, regroup, or move queued work within latency targets.
Request-scoped state
The request releases or returns its resources when it ends.
Clock-driven
A session advances on every tick
For Moshi, each tick spans 80 ms. Overruns consume available playback-buffer margin.
Session-scoped state
State stays attached to the live session and is needed again at the next tick.
Room to trade latency for efficiencyBrief waiting can improve utilization; that delay is charged to request latency.
A recurring timing contractThe repeated path must keep pace with the 80 ms cadence to avoid drawing down the playback buffer.
This distinction is not about text versus audio. Offline speech synthesis can wait in a queue. Live transcription, control loops, and duplex conversation cannot. What matters is whether the output is attached to a recurring wall-clock deadline.
Why a cascade behaves differently
A conventional voice agent is usually a cascade: speech recognition feeds an LLM, which feeds text-to-speech. A voice-activity detector or turn detector decides when the user has finished speaking and triggers the answering stages.
Parts of that pipeline may stream, and the recognizer may run continuously. But the expensive generative work is still organized around turns. Between responses, the LLM and synthesizer can be idle; while answering, the synthesizer can generate ahead into a buffer. The dialogue model does not owe a step on every frame of the entire call.
A duplex model removes that trigger. It listens and generates simultaneously, including while its own output is silence. That silence is not empty time: its duration helps the model distinguish a hesitation from the end of a turn, decide whether to backchannel, and react to an interruption.
Skipping silent frames would also remove elapsed time from the sequence. Unless the model was designed for missing or aggregated frames, this is not a transparent serving optimization.
Batching survives, but capacity changes
It is easy to draw the wrong conclusion from a long-lived session: that one call monopolizes one GPU. It does not.
At each tick, the engine can batch compatible sessions whose next steps are due together. The model weights are shared across the sessions in the batch, just as they are in conventional batched decoding. Batching remains essential to the economics of duplex inference.
The strongest public evidence is an experimental vLLM-Omni path for PersonaPlex 7B that advanced 32 session states in 70.2 ms against an 80 ms tick on a 140 GB-class GPU.
It proves that duplex can batch, but not that 32 production calls are sustainable: only four-session end-to-end concurrency was validated, and no long-running tail measurement was published.
For these models, the useful output rate of one conversation is imposed by the clock. Faster execution does not make that conversation produce more useful frames; it creates the headroom to run more conversations without missing them. Tokens per second is therefore the wrong headline metric.
The limiting question for current implementations is:
How many concurrent sessions can one GPU sustain while keeping every session's frames on schedule?
A credible answer has to identify the model and GPU, time the complete path over realistic call lengths, and report the frames delivered on time for each session. The system's capacity is the highest concurrency at which that contract continues to hold.
Running one conversation in real time is necessary; maintaining the guarantee at economically useful concurrency is the unsolved serving problem.
What the clock changes
Once the workload is defined as a recurring deadline rather than a queue of requests, several familiar serving assumptions stop transferring cleanly.
1. The clock closes the batch
A request scheduler can wait briefly for more work if doing so improves utilization. A duplex scheduler cannot wait past the next frame deadline. At every tick, it must run the sessions that are due with the batch it has.
Figure 02 · Batching on a clock
The clock closes the batch.
Compatible sessions due on the same tick share a batched execution path and deadline.
Due sessions → Batch
Live state + current frame
Complete model step
Model-specific components; schematic widths
- Audio
encoder - BackboneShared weights
Batched sessions - Dependent
decoder(s) - Codec
+ state
Resident session state
Each session keeps its own state between ticks and re-enters a compatible batch when its next frame is due.
Measured capacity
For a given model and GPU, capacity is the highest concurrency sustained over realistic call lengths while every session remains on schedule.
The scheduler can still decide where a new session should live. What it loses is freedom over when that session's next step runs. Placement and admission therefore become more important than queue reordering.
This also changes autoscaling. Starting another replica after latency has already deteriorated may be too late: existing sessions carry state and cannot be moved cheaply in the middle of a conversation. Capacity has to be available in the right region before the next call is admitted.
2. Overload becomes correlated
When sessions share one model step, an overrun makes all of them late together. This is different from a queued system, where one old request can be slow while newer requests remain unaffected.
The danger is not only a single slow step, but a run of them. A 30 ms buffer can absorb one 20 ms overrun. It cannot absorb four consecutive overruns of the same size.
For that reason, mean latency is not a sufficient capacity metric. The distribution, the longest run of missed ticks, and the remaining buffer margin all matter.
There is a second failure mode: one session can fall behind while the rest of the batch remains healthy because its input was late, its output queue stalled, or its state was recycled incorrectly.
Fleet-wide averages dilute that failure. A single broken call among 32 moves the mean only slightly, even though that caller hears every gap.
3. State remains live for the call
Conventional serving engines can reclaim, evict, or recompute KV state as requests finish or wait. A duplex session needs its state again on the next tick. Moving or rebuilding it spends part of a deadline that is already short.
The state need not grow forever. Moshi, for example, uses a context of 3,000 steps (about four minutes at 12.5 Hz). But the cache remains attached to the session until the call ends, and its memory traffic is paid repeatedly.
How that cost evolves depends on the implementation. A fixed ring can reserve and process the full window from the start. A length-aware or paged implementation can process only the populated history, but its cost grows as the call gets older.
In either case, admission cannot be based only on the memory used by a new call at its first tick.
4. Ordinary metrics can stay green while a call fails
Request-serving dashboards focus on queue time, time to first token, inter-token latency, throughput, and request completion. Duplex serving needs those metrics, but they do not identify whether each audio frame reached its session on time.
The transport layer can obscure the failure further. A runtime may send silence when the model has not produced a frame, while the network continues to deliver packets at a regular cadence. From the transport's perspective, the stream is healthy. From the caller's perspective, the assistant has stopped responding.
The serving layer therefore has to record, for every session and every tick, whether the input arrived on time, whether the model ran on real or substituted input, whether the output was generated or filled in, and how far the session has fallen behind. Without that provenance, intentional silence and missed computation look the same.
A continuous inference engine
These requirements point to a different execution and control stack. It must optimize the complete repeated path (including encoders, the backbone, dependent decoders, codecs, state transitions, and data movement) while retaining session state, closing each batch on the clock, admitting calls against measured headroom, and tracking every session's deadlines.
Paged KV caches, CUDA graphs, kernel fusion, and batching remain useful. Reaching useful session density will also require new execution techniques that fuse more of the repeated path and keep more of its control on the GPU.
The difference is the objective: the execution layer and scheduler are optimized together to maximize the number of sessions that remain on time.
Duplex voice makes this requirement audible, but it extends to continuous inference more broadly. The exact policy may change (another workload may drop frames or use a freshness bound instead of a fixed tick), but the abstraction remains a persistent session with an explicit timing and state contract.
For conventional inference, the scheduler asks which request should run next. For continuous inference, it already knows when the work must run.
The unit of work is the session. The constraint is the clock.