The result in brief
4,800 live transcription streams, sharing one GPU and staying on schedule.
Nemotron 3.5 ASR Streaming transcribes speech as it arrives, in 80 ms chunks. Keeping a transcript live means returning every chunk of every stream before the next one lands, at whatever number of streams the GPU holds.
The .wave inference engine compiles the whole model into one persistent GPU program and advances all of its streams together each period. On one H100, it served 4,800 concurrent streams with every frame completed across three runs, measured where a caller would measure it.
Against the 240 concurrent streams NVIDIA's model card lists for this setting on one H100, the measured density gives 20× more streams per GPU and approximately 95% lower GPU cost per stream at equal GPU price and utilization. Transcripts read within 0.23 percent of the fp32 reference, at the precision the production NIM runs.
01 / The challenge
Capacity that keeps pace with speech.
A live transcription service must return each 80 ms of audio before the next arrives, for every stream it holds. Adding streams is useful only while their words still come back on time.
The model's own streaming encoder carries a cache per stream and reads a small chunk of new audio each step; the work per step is small, and what limits a GPU is how many such steps it can complete together inside the period, with their caches, their decoders and their input and output paths, without leaving the device for each one.
For the operator of a transcription service, the streams one GPU holds on time set the GPU cost of every transcript. The model card lists 240 at this setting.
02 / The solution
Run the whole model together.
.wave compiles the front end, the encoder and the decoder into one execution program that processes a fixed cohort of streams together. Model weights are read once for the whole cohort, every stream's cache stays resident between chunks, and the GPU advances through the model without returning to the CPU for each operation.
The data plane feeds the program from the streams' WebSockets and returns each stream's words as they are emitted, so the whole path from a caller's audio to its transcript runs at the model's own cadence.
03 / Business impact
More transcripts from the same GPU hour.
Capacity per GPU
The qualified .wave profile serves 4,800 concurrent streams on one H100, against the 240 the model card lists for the same setting. This multiplies the live streams that can share a given GPU allocation.1
GPU cost per stream
At the same hourly GPU price and utilization, dividing the GPU hour among 4,800 streams instead of 240 reduces the GPU-cost contribution per stream by 95%.2
Price per minute of audio
Published rates for the same model id, checked 16 September 2026.
Streams stay on time
At 4,800 concurrent streams, every one of 7,800,000 frames per run completed, in each of three runs, and no stream's completion cadence left its bound. Transcripts read within 0.23 percent of the fp32 reference. This is what determines usable capacity: how many streams can share a GPU while their words keep arriving on schedule.3
7,800,000 frames per run
4,800 streams · three runs
04 / Under the hood
A compiled program that stays on the GPU.
At the core of the .wave engine is WPK, the Wave Persistent Kernel. The engine's compiler generates the graph, schedule and memory plan from Python Tensor IR and executes the model as one persistent GPU program, with shared weight reads and coordination between operations kept on the device. The same compiler and engine serve the full-duplex Nemotron VoiceChat model; what was specific to this model was measured in days.
Profiles are compiled for a model and a stream capacity. The technical article explains the compiler, memory layout, worker coordination and the measurement protocol in detail.
Read the WPK technical articleMeasurement notes
- Density. Nemotron 3.5 ASR Streaming 0.6B on one H100 SXM 80 GB at the 80 ms chunk, 4,800 concurrent streams measured at a benchmark client under a preregistered protocol, compared with the 240 concurrent streams NVIDIA's model card lists for the same setting on one H100. The card does not state its measurement method; the comparison is against that published figure, not against a matched measured run of the NIM.
- Price. $0.00045 per minute of audio, against the $0.0045 a minute Together AI publishes for
nvidia/nemotron-3.5-asr-streaming-0.6b, the same weights, checked 16 September 2026. The gap is the density passed through: one H100 holds roughly twenty times the concurrent streams, so the same margin yields roughly a tenth of the price. Our rate is for live streaming, where a slot is held open for the length of the call; transcribing recorded files is a cheaper product everywhere, since it runs faster than real time and holds nothing open, and DeepInfra sells file transcription of these same weights at $0.0002 per minute. Published rates change; this comparison is dated. - Frames. A frame is 80 ms of one stream's audio and has one completion. At 4,800 streams, .wave recorded 7,800,000 completed frames of 7,800,000 in each of three runs; the per-stream p99 interval between completions stayed under the 120 ms bound for every stream (worst stream 106.3 ms), and the interval from a frame's release to its completion was 207 to 211 ms at the median with the jitter hold at one period. Transcripts were scored as word error rate against the fp32 reference: 0.23 percent mean, 1.34 percent worst over 23 programs.
Three consecutive runs at the qualified point, with the same weights, precision and recorded programs; the encoder in BF16 with FP32 accumulation and the decoder in FP32, the precision the production NIM runs. Timing at the benchmark client on the serving host over loopback, through the gateway's signed ticket and the data plane's WebSocket; wide-area transport is excluded. The host provider's periodic GPU preemption windows were recorded and none fell inside the runs.
Measurement data