Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2
NVIDIA CUDA Multi-Process Service (MPS) combined with Triton Inference Server on Amazon EC2 GPU instances reduced GPU infrastructure requirements by 75% (from 16 instances to 4) for Heidi Health's clinical ASR workload A single ASR inference request typically uses only 15–20% of an NVIDIA L40S GPU's compute capacity; default CUDA time-slicing leaves ~80% of hardware idle with sequential access and context-switch overhead The hybrid inference pipeline runs the Conformer encoder through ONNX Runti
Analysis
TL;DR
- NVIDIA CUDA Multi-Process Service (MPS) combined with Triton Inference Server on Amazon EC2 GPU instances reduced GPU infrastructure requirements by 75% (from 16 instances to 4) for Heidi Health's clinical ASR workload
- A single ASR inference request typically uses only 15–20% of an NVIDIA L40S GPU's compute capacity; default CUDA time-slicing leaves ~80% of hardware idle with sequential access and context-switch overhead
- The hybrid inference pipeline runs the Conformer encoder through ONNX Runtime with TensorRT EP (FP16 precision, kernel fusion) while the RNN-T decoder runs natively in PyTorch CUDA with CUDA graph caching
- MPS enables concurrent kernel execution across multiple processes sharing a single GPU context, eliminating context-switching overhead and supporting configurable SM partitioning via CUDA_MPS_ACTIVE_THREAD_PERCENTAGE
- Transcription instances use 25% SM allocation with 4 concurrent processes (~2.5 GB VRAM each); diarization instances use 12% SM with 8 concurrent processes (~1.8 GB each), achieving 92.1 RPS per GPU with sub-second latency
Why It Matters
This case study demonstrates a practical, production-proven approach to dramatically reducing GPU inference costs for latency-sensitive ASR workloads without requiring model retraining or application code changes. For AI practitioners running small-to-medium models that underutilize GPUs, the combination of MPS, TensorRT optimization, and Triton batching offers a replicable blueprint for achieving significant infrastructure savings while maintaining strict latency SLAs.
Technical Details
- CUDA MPS (Multi-Process Service): A binary-compatible CUDA API implementation that funnels all processes through a single GPU context managed by an MPS daemon, enabling concurrent kernel execution on different SMs without context-switch overhead. Memory protection is maintained via separate address spaces per client process.
- Hybrid ONNX/TensorRT + PyTorch pipeline: The compute-heavy 24-layer Conformer encoder (1024 hidden dimensions) is converted to ONNX and executed via TensorRT EP with FP16 precision calibration and operator fusion. The RNN-T Token-and-Duration Transducer decoder remains in native PyTorch CUDA, leveraging CUDA graph caching for variable-length token generation flexibility.
- Triton Inference Server batching: Dynamic batching with a 50 ms accumulation delay and preferred batch sizes of [4, 8, 16] is used for transcription. This allows the scheduler to form optimally sized batches that maximize GPU throughput while respecting latency constraints.
- MPS configuration: Two separate MPS deployments on dedicated GPU instances — transcription instances allocate 25% SM per process with 4 concurrent processes (~2.5 GB VRAM each on L40S 48 GB), and diarization instances allocate 12% SM with 8 concurrent processes (~1.8 GB each).
- Performance results: The optimized setup achieves 92.1 RPS per GPU with mean latency < 650 ms and p99 < 1,000 ms, reducing Heidi Health's fleet from 16 GPU instances to 4 while processing 2.4 million clinical consultations weekly across 190 countries.
Industry Insight
- For workloads where individual inference requests underutilize GPU compute (common with smaller models like Parakeet TDT 0.6B), MPS should be evaluated before scaling out to additional GPU instances — it can deliver 75%+ cost reductions without model changes or retraining.
- Hybrid optimization strategies that apply TensorRT to static compute kernels (encoders) while preserving framework flexibility for dynamic components (decoders with variable-length generation) offer a pragmatic middle ground between full compilation and native execution.
- The three GPU sharing mechanisms (time-slicing, MIG, MPS) serve distinct use cases: MIG for hard isolation in multi-tenant environments, MPS for high-concurrency small-model workloads, and time-slicing only for simple single-model deployments — choosing the right mechanism is critical to cost-performance optimization.
Disclaimer: The above content is generated by AI and is for reference only.