Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware
First reproducible hardware-level energy benchmark measuring GPU power draw of nine open-source LLMs (1B–7B parameters) on consumer RTX 4060Ti 16GB hardware using Ollama gemma3:1b and llama3.2:1b achieve the lowest energy cost at 0.56 J/token and 0.65 J/token respectively, with throughput exceeding 170 tok/s 7B-Mistral consumes up to 4.4x more energy per token than the most efficient model, demonstrating that parameter count alone is a poor predictor of energy efficiency qwen3.5:2b shows anomalo
Analysis
TL;DR
- First reproducible hardware-level energy benchmark measuring GPU power draw of nine open-source LLMs (1B–7B parameters) on consumer RTX 4060Ti 16GB hardware using Ollama
- gemma3:1b and llama3.2:1b achieve the lowest energy cost at 0.56 J/token and 0.65 J/token respectively, with throughput exceeding 170 tok/s
- 7B-Mistral consumes up to 4.4x more energy per token than the most efficient model, demonstrating that parameter count alone is a poor predictor of energy efficiency
- qwen3.5:2b shows anomalously high per-prompt energy due to extended internal reasoning, revealing a critical gap in current efficiency metrics that fail to distinguish between token generation modes
- Model architecture and quantization strategy are identified as primary drivers of energy efficiency, not just raw parameter count
Why It Matters
As local LLM deployment grows for privacy and on-premise inference, practitioners need concrete data on energy costs to make informed hardware and model selection decisions—yet the field has overwhelmingly focused on accuracy benchmarks while neglecting power consumption. This paper fills that gap by providing reproducible, hardware-level measurements that directly inform the total cost of ownership for edge and consumer-grade AI deployments.
Technical Details
- Benchmark setup: Nine open-source LLMs ranging from 1B to 7B parameters evaluated on a single NVIDIA RTX 4060Ti 16GB GPU using the Ollama inference engine, with GPU power draw sampled at 2Hz via nvidia-smi across a fixed prompt set
- Metrics measured: Mean and peak power consumption, total energy per prompt (J/prompt), energy per output token (J/token), and throughput (tok/s)—providing a multi-dimensional view of efficiency beyond single-metric benchmarks
- Key findings: gemma3:1b (0.56 J/token) and llama3.2:1b (0.65 J/token) led in energy efficiency with >170 tok/s throughput; 7B-Mistral consumed up to 4.4x more energy per token than gemma3:1b despite only ~7x more parameters, confirming architecture and quantization dominate over parameter count
- Anomaly detection: qwen3.5:2b exhibited unexpectedly high per-prompt energy attributed to extended internal reasoning processes, highlighting that current benchmarking methodologies do not adequately account for different token generation modes (e.g., chain-of-thought vs. direct response)
Industry Insight
- Organizations deploying LLMs on consumer or edge hardware should prioritize architecture and quantization choices over raw model size when optimizing for energy efficiency; a 1B-parameter model can outperform a 7B model on both energy and throughput depending on design choices
- Benchmarking frameworks and evaluation pipelines must incorporate energy-per-token metrics alongside accuracy and latency to reflect real-world operational costs, especially as sustainability becomes a regulatory and business priority
- The qwen3.5:2b anomaly suggests that models with extended reasoning capabilities may appear inefficient under standard benchmarks, indicating a need for mode-aware evaluation protocols that distinguish between reasoning-intensive and direct-generation inference patterns
Disclaimer: The above content is generated by AI and is for reference only.