Measuring Performance of Transformer Inference
LLM inference optimization requires precise measurement across multiple dimensions: latency, throughput, memory, and cost per token, as optimizing one metric can degrade another Time to first token (TTFT) and time per output token (TPOT) are critical user-facing metrics that must be tracked separately from aggregate latency Prefill and decode phases have fundamentally different performance characteristics and should be measured independently, as they stress different hardware bottlenecks Tail la
Analysis
TL;DR
- LLM inference optimization requires precise measurement across multiple dimensions: latency, throughput, memory, and cost per token, as optimizing one metric can degrade another
- Time to first token (TTFT) and time per output token (TPOT) are critical user-facing metrics that must be tracked separately from aggregate latency
- Prefill and decode phases have fundamentally different performance characteristics and should be measured independently, as they stress different hardware bottlenecks
- Tail latency (p90, p95, p99) is essential for understanding real-world user experience, not just mean or median performance
- CUDA events and proper synchronization are necessary for accurate GPU inference benchmarking, and
time.perf_counter()is recommended overtime.time()for Python-based measurements
Why It Matters
This guide addresses a fundamental gap in LLM deployment: without rigorous performance measurement, practitioners risk optimizing for the wrong metrics, leading to systems that appear faster in benchmarks but deliver worse user experience. As LLM services scale, understanding the tradeoffs between TTFT, TPOT, throughput, and cost per token directly impacts both user satisfaction and operational economics.
Technical Details
- Core Metrics: Latency (total request time), TTFT (wall time until first token), TPOT (average inter-token generation time), throughput (tokens/requests per second), memory usage, accelerator utilization, and cost per token
- Prefill vs. Decode Separation: The article provides a Python implementation using Hugging Face transformers that manually separates prefill (processing the full prompt) from decode (iterative token generation with KV cache), avoiding the
generate()method to expose both phases independently - Tail Latency Reporting: Recommends reporting p90, p95, and p99 percentiles alongside mean/median using NumPy, with a utility function that prevents the common pitfall of optimizing averages while worsening worst-case scenarios
- GPU Benchmarking: Highlights the need for CUDA events and
torch.cuda.synchronize()to ensure accurate GPU timing, as GPU operations are asynchronous and wall-clock measurements without synchronization can be misleading - Multi-request and Multi-GPU Considerations: The chapter structure indicates coverage of concurrent request benchmarking, multi-GPU setups, and multi-machine distributed inference, along with cost-per-token calculations for economic evaluation
Industry Insight
- Organizations should establish standardized benchmarking pipelines that report both head-line metrics (mean latency, throughput) and tail metrics (p99) before and after any optimization, as many published "speedups" only improve averages while degrading worst-case latency
- The prefill/decode separation is critical for capacity planning: prefill-bound workloads (long prompts, short outputs) and decode-bound workloads (short prompts, long outputs) require different hardware configurations and scaling strategies
- Cost-per-token metrics should be integrated into performance dashboards alongside latency numbers, as the most "performant" optimization may be economically unsustainable at scale; this bridges the gap between engineering and business stakeholders
Disclaimer: The above content is generated by AI and is for reference only.