Speculative Decoding on CPUs: Nearly 4x Faster Token Generation with DFlash
DFlash speculative decoding is now supported in vLLM v0.25.0 for CPU inference, enabling lossless acceleration without changing model output distribution Testing with Qwen3.5-9B on Intel Xeon 6-powered AWS r8i instances achieved a 3.92x throughput increase at concurrency 1, translating to a 74% cost reduction per generated token DFlash uses a block-diffusion drafter that predicts an entire block of tokens in one pass with bidirectional attention, combined with target model hidden feature injecti
Analysis
TL;DR
- DFlash speculative decoding is now supported in vLLM v0.25.0 for CPU inference, enabling lossless acceleration without changing model output distribution
- Testing with Qwen3.5-9B on Intel Xeon 6-powered AWS r8i instances achieved a 3.92x throughput increase at concurrency 1, translating to a 74% cost reduction per generated token
- DFlash uses a block-diffusion drafter that predicts an entire block of tokens in one pass with bidirectional attention, combined with target model hidden feature injection into the draft model's KV cache
- Acceptance rates vary by domain: higher for structured prompts (code, math) and lower for conversational or out-of-domain use cases
- The optimal
num_speculative_tokensparameter depends on the target/draft model pair, dataset, concurrency, and hardware characteristics
Why It Matters
Speculative decoding on CPU represents a significant milestone for making high-performance LLM inference accessible beyond GPU-only deployments, potentially democratizing cost-effective AI serving. The 3.92x speedup with DFlash demonstrates that CPU-based inference can now compete meaningfully with traditional autoregressive baselines, offering a practical path for organizations looking to reduce cloud infrastructure costs. For AI practitioners, this means speculative decoding is no longer an GPU-exclusive optimization and can be deployed on widely available CPU instances.
Technical Details
- DFlash Architecture: Developed by Z Lab, DFlash uses a small block-diffusion drafter that predicts all masked positions in a single forward pass with bidirectional attention within the block, rather than generating draft tokens serially. Hidden features from the target model are injected into each draft layer's KV cache to improve draft quality without requiring the drafter to reconstruct full context independently.
- Lossless Acceleration: Unlike quantization or other lossy optimizations, speculative decoding uses rejection sampling to recover the exact target distribution, ensuring zero accuracy degradation.
- Hardware Optimization for Xeon: At small batch sizes, DFlash transforms per-token matrix-vector operations (relying on Intel AVX-512) into matrix-matrix operations accelerated by Intel AMX, reusing target weights across multiple candidate positions and trading spare compute for saved memory bandwidth.
- Benchmark Results (Qwen3.5-9B, concurrency 1, output length 128): GSM8K achieved 4.16x speedup (75.98% acceptance rate), HumanEval achieved 3.99x speedup (74.93% acceptance rate), and MT-Bench showed strong gains as well. Average across all three datasets: 3.92x throughput improvement.
- Deployment: Enabled via a single vLLM configuration flag (
--speculative-config) specifying the DFlash method, draft model (z-lab/Qwen3.5-9B-DFlash), andnum_speculative_tokens(tested at 15).
Industry Insight
- Organizations running LLM workloads on CPU infrastructure can achieve near-4x throughput improvements with zero accuracy loss, making CPU-based serving a viable cost-optimization strategy alongside GPU deployments.
- Practitioners should align their benchmarking datasets with their target model's domain—structured tasks like code and math show higher acceptance rates and greater speedups, while conversational workloads may see diminished returns; domain-specific models require matching drafters for optimal results.
- The availability of pre-trained drafters through Z Lab's DFlash collection and Red Hat AI's Speculator library lowers the barrier to adoption, but teams should expect to tune
num_speculative_tokensand other parameters per workload rather than relying on defaults.
Disclaimer: The above content is generated by AI and is for reference only.