How Much Does a Local LLM Actually Cost to Run? I Measured Every Watt on Apple Silicon
Cost per token is driven by throughput and power draw, not parameter count. Larger mixture-of-experts (MoE) models can be cheaper and faster to run than smaller dense models due to reduced data movement per token. The 27B dense model is the most expensive to run despite being mid-sized, because it activates all parameters per token, maximizing memory bandwidth usage and power consumption. MoE models activate only a subset of parameters per token, reducing bytes-per-token and improving efficiency
Analysis
TL;DR
- Cost per token is driven by throughput and power draw, not parameter count.
- Larger mixture-of-experts (MoE) models can be cheaper and faster to run than smaller dense models due to reduced data movement per token.
- The 27B dense model is the most expensive to run despite being mid-sized, because it activates all parameters per token, maximizing memory bandwidth usage and power consumption.
- MoE models activate only a subset of parameters per token, reducing bytes-per-token and improving efficiency.
- Quantization further lowers cost by shrinking the footprint of weights in memory.
Why It Matters
This analysis challenges the assumption that larger models are inherently more expensive to run locally. For AI practitioners deploying LLMs on consumer hardware, selecting models based solely on parameter count can lead to suboptimal choices. Understanding how architecture (dense vs. MoE), quantization, and memory bandwidth interact with energy cost enables smarter deployment decisions that balance performance, cost, and scalability.
Technical Details
- Measurement tool: TokenWatt, an OpenAI-compatible proxy for Apple Silicon that measures marginal energy cost via Apple’s IOReport interface, calibrated against a wall-metered Shelly Plug US Gen4 (±2.6% to ±4.5% error).
- Hardware: M3 Ultra Mac Studio with 96 GB unified memory (no discrete GPU), running sustained generation loops at three durations (120s, 360s, 720s), three passes each, with 15-minute idle baselines between runs.
- Models tested: Qwen3.5-4B (dense, 8-bit), Qwen3.6-35B-A3B (MoE, 3B experts, ~4–6 bit), Qwen3-Coder-Next (~80B MoE, 3B experts, ~4–6 bit), gpt-oss-120b (120B MoE, 5B experts, ~4–6 bit), Qwen3.6-27B (dense, 8-bit).
- Key metric: Cost per million output tokens calculated as (power draw in watts × electricity rate $0.31/kWh) / (tokens per second), normalized to per-million-token basis.
- Throughput bottleneck: Single-stream decode speed limited by memory bandwidth — dense models stream all weights per token; MoE models route only active experts, drastically reducing data movement.
Industry Insight
When deploying local LLMs on shared-memory or bandwidth-constrained systems like Apple Silicon, prioritize mixture-of-experts architectures over dense ones even if they have higher total parameter counts — they often deliver lower cost per token and higher throughput. Quantization should be applied aggressively where acceptable, as it compounds efficiency gains in both memory footprint and power. Developers and enterprises should benchmark real-world energy costs using calibrated tools rather than relying on theoretical FLOPs or parameter counts when evaluating inference economics.
Disclaimer: The above content is generated by AI and is for reference only.