Request-Level Energy Attribution for Batched LLM Serving
Introduces JouleShare, a framework for request-level energy attribution in batched LLM serving, addressing the gap between aggregate GPU telemetry and per-request energy accounting needs Uses an offline harness with vLLM to establish ground-truth Shapley energy values by replaying request subsets and integrating GPU power telemetry Proposes JCalib, a lightweight calibration model that predicts Shapley shares from inexpensive request features for real-time serving use Token-proportional attributi
Analysis
TL;DR
- Introduces JouleShare, a framework for request-level energy attribution in batched LLM serving, addressing the gap between aggregate GPU telemetry and per-request energy accounting needs
- Uses an offline harness with vLLM to establish ground-truth Shapley energy values by replaying request subsets and integrating GPU power telemetry
- Proposes JCalib, a lightweight calibration model that predicts Shapley shares from inexpensive request features for real-time serving use
- Token-proportional attribution deviates significantly from exact Shapley (0.440–0.458 normalized L1), but JCalib reduces error to 0.116 (static) and 0.177 (continuous batching)
- Demonstrates that token-based attribution is unreliable under batched execution and that measured Shapley ground truth can effectively calibrate low-cost features for fairer energy allocation
Why It Matters
This work addresses a critical gap in sustainable AI infrastructure: as LLM serving scales, accurate per-request energy accounting becomes essential for chargeback models, sustainability reporting, and workload optimization. The findings challenge the common industry assumption that token count is a sufficient proxy for energy cost, which has direct implications for pricing models and carbon accounting in production LLM systems.
Technical Details
- JouleShare framework: Combines an offline energy attribution harness with an online calibration model. The harness replays request subsets under vLLM with a reproducible protocol, integrates GPU power telemetry, and computes exact Shapley energy values for each request.
- JCalib calibration model: A lightweight model trained on Shapley ground truth to predict per-request energy shares from cheap-to-compute features (e.g., input/output token counts, batch position), enabling real-time attribution during serving.
- Benchmarking across 16 model/workload combinations: Evaluated on three data-center GPUs under both static and continuous batching strategies, providing robust empirical validation.
- Quantitative results: Token-proportional attribution shows normalized L1 errors of 0.440 (static batching) and 0.458 (continuous batching) versus exact Shapley. JCalib reduces these to 0.116 and 0.177 respectively, outperforming a standalone-measurement baseline that is impractical for online use.
- Sampled Shapley extension: Used to scale measurements to larger group sizes where exact Shapley becomes computationally prohibitive, confirming that the attribution gap persists and a single offline calibration remains the most accurate deployable approach.
Industry Insight
- Cloud providers and LLM API operators should reconsider token-based pricing models, as they systematically misallocate energy costs under batched serving; adopting calibrated attribution like JCalib could enable fairer chargeback mechanisms.
- Sustainability and carbon-accounting teams can leverage measured Shapley ground truth to validate and improve existing attribution rules, ensuring compliance with emerging ESG reporting standards.
- The offline-harness-then-calibrate paradigm demonstrated here offers a practical template for other resource-attribution problems (e.g., memory, latency) where exact per-request accounting is infeasible online but valuable for analytics and billing.
Disclaimer: The above content is generated by AI and is for reference only.