I Benchmarked Every MTP Depth on Qwen3.8–27B-UD-Q4. Here You Are.
MTP on Qwen3.8-27B-UD-Q4 achieves approximately 2× throughput improvement on an RTX 4090, with K_M quant jumping from 48.5 to 96.2 t/s and K_XL from 46.3 to 84.2 t/s at optimal draft depth The optimal draft_num_predict value is 5 across both quantizations, delivering the best performance on K_M, K_XL, average, and worst-case scenarios K_XL degrades below baseline at draft depth 8, while K_M remains viable until depth 11, demonstrating that quantization choice directly determines the safe upper b
Analysis
TL;DR
- MTP on Qwen3.8-27B-UD-Q4 achieves approximately 2× throughput improvement on an RTX 4090, with K_M quant jumping from 48.5 to 96.2 t/s and K_XL from 46.3 to 84.2 t/s at optimal draft depth
- The optimal draft_num_predict value is 5 across both quantizations, delivering the best performance on K_M, K_XL, average, and worst-case scenarios
- K_XL degrades below baseline at draft depth 8, while K_M remains viable until depth 11, demonstrating that quantization choice directly determines the safe upper bound for speculative decoding
- MTP is a VRAM-for-speed tradeoff, not a quality tradeoff: verification ensures output fidelity remains identical to normal decoding
- The useful drafting range is narrow (2–5 tokens), with excessive drafting (N>8 for K_XL, N>11 for K_M) causing severe slowdowns up to 5.3× worse than baseline despite all weights remaining GPU-resident
Why It Matters
This benchmark provides the first empirical evidence that the optimal MTP draft depth is not a universal constant but shifts significantly based on quantization choice, directly challenging the vague "keep it low" advice found across documentation. For AI practitioners running 27B-class models locally, these findings mean that deploying MTP without hardware-specific tuning risks catastrophic performance degradation rather than improvement. The work also clarifies the fundamental bottleneck in local inference—memory bandwidth, not compute—and demonstrates how speculative decoding exploits that constraint when configured correctly.
Technical Details
- Architecture: Qwen3.8-27B-UD-Q4 with Unsloth-trained MTP head (Q4_0, ~1.37 GB), tested across two quantization variants: UD-Q4_K_M (17 GB) and UD-Q4_K_XL (18 GB), both with 96K context window and q8_0 KV cache
- Hardware: Intel Core i7-14700KF, 128 GB RAM, NVIDIA RTX 4090 (24 GB VRAM), Ubuntu 24.04, Ollama runtime with Flash Attention enabled, OLLAMA_NUM_PARALLEL=1
- Benchmark methodology: 96 total runs (16 draft depths × 2 quants × 3 runs), using a standardized instruction prompt requesting continuous technical prose about transformer decoding; throughput measured as completion tokens per second excluding prefill and model load, with 16-token warmup, shuffled model order, and adaptive stopping criteria ensuring equal successful runs across configurations
- Key findings: Peak speedup at draft_num_predict=5 (K_M: 96.2 t/s, K_XL: 84.2 t/s); K_XL breaks even at N=8 (8.7 t/s vs 46.3 baseline at N=15, a 5.3× slowdown); K_M tolerates up to N=11 before degrading; both quants show identical peak depth but divergent failure thresholds due to the ~1 GB weight difference
- Implementation: MTP configured via Ollama Modelfile with DRAFT instruction pointing to separate Q4_0 module; sampling parameters held constant (temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repeat_penalty 1.0)
Industry Insight
- practitioners should treat draft_num_predict as a hardware- and quantization-specific parameter requiring empirical calibration rather than adopting a one-size-fits-all value from documentation; even a 1 GB quantization difference can shift the safe upper bound by 3 draft levels on identical hardware
- MTP speculative decoding delivers genuine 2× speedups for local 27B-class inference without quality loss, but the narrow optimal window (2–5) means misconfiguration risks severe slowdowns; users should benchmark their specific quantization before deployment rather than assuming safety from generic recommendations
- the VRAM cost of the MTP head (~1.37 GB) becomes significant on 24 GB cards when combined with heavier quantizations and large context windows, creating a hard constraint on how much headroom exists for speculative decoding; this tradeoff will intensify as model sizes grow and context windows expand, making quantization selection a strategic decision beyond mere quality considerations
Disclaimer: The above content is generated by AI and is for reference only.