Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices
The paper presents a runtime-aware latency prediction framework for deploying large language models (LLMs) on heterogeneous edge devices, addressing the complexity of inference latency influenced by model architecture, prompt behavior, runtime backend, hardware utilization, dynamic voltage and frequency scaling (DVFS), and thermal variation. The framework separates inference into prefill and decode phases and adaptively fuses static descriptors with dynamic hardware telemetry through a gated pre
Analysis
TL;DR
- The paper presents a runtime-aware latency prediction framework for deploying large language models (LLMs) on heterogeneous edge devices, addressing the complexity of inference latency influenced by model architecture, prompt behavior, runtime backend, hardware utilization, dynamic voltage and frequency scaling (DVFS), and thermal variation.
- The framework separates inference into prefill and decode phases and adaptively fuses static descriptors with dynamic hardware telemetry through a gated prediction model, improving latency prediction accuracy across different devices.
- Evaluation on Pixel mobile devices and validation on Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU platform show significant improvements in latency prediction accuracy, with cross-device calibration enhancing the framework's transferability.
- The results highlight the importance of runtime-aware prediction and lightweight calibration for reducing profiling costs and supporting latency-aware LLM deployment across heterogeneous edge platforms.
Why It Matters
Accurate latency prediction is crucial for deploying LLMs on edge devices, where performance can vary significantly due to hardware and runtime differences. This paper addresses a critical challenge in the field by providing a framework that can adapt to different devices and predict latency more accurately, which is essential for optimizing LLM deployment in real-world scenarios.
Technical Details
- The framework represents each inference request as a hardware-runtime-model-prompt configuration, capturing the complex interactions between these factors.
- Inference is divided into prefill and decode phases, allowing for more granular and accurate latency predictions.
- A gated prediction model is used to fuse static descriptors (e.g., model architecture) with dynamic hardware telemetry (e.g., DVFS, thermal variation), enhancing the adaptability of the prediction.
- The framework is evaluated on Pixel mobile devices and validated on a range of edge devices, including Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU, demonstrating its effectiveness across different hardware platforms.
- Cross-device calibration is shown to significantly improve prediction accuracy, making the framework more transferable and practical for real-world deployment.
Industry Insight
- The runtime-aware latency prediction framework can significantly reduce the profiling cost and time required for deploying LLMs on edge devices, making it more feasible for businesses to optimize their LLM deployments.
- The ability to adapt to different hardware and runtime conditions through lightweight calibration can lead to more efficient and effective LLM deployment strategies, especially in heterogeneous environments.
- This research highlights the importance of considering runtime and hardware factors in LLM deployment, which can guide future developments in edge AI and contribute to more robust and scalable LLM solutions.
Disclaimer: The above content is generated by AI and is for reference only.