TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
TreeGraft introduces a multi-drafter framework for tree-based speculative decoding, combining drafters of different computational costs to jointly construct a shared draft tree The stronger drafter rescoring mechanism updates candidate scores, reselects grafting positions, and recovers promising paths left unexplored by the weaker drafter Non-destructive integration of stronger drafter expansions preserves existing branches that may still be accepted by the target model A lightweight scheduler,
Analysis
TL;DR
- TreeGraft introduces a multi-drafter framework for tree-based speculative decoding, combining drafters of different computational costs to jointly construct a shared draft tree
- The stronger drafter rescoring mechanism updates candidate scores, reselects grafting positions, and recovers promising paths left unexplored by the weaker drafter
- Non-destructive integration of stronger drafter expansions preserves existing branches that may still be accepted by the target model
- A lightweight scheduler, distilled from an offline value system, dynamically decides when to invoke the stronger drafter to control drafting cost
- TreeGraft outperforms the better of two fixed single-drafter strategies by 15.1% on average across 10 model pairs and 6 benchmarks, with a maximum gain of 26.6%
Why It Matters
TreeGraft addresses a fundamental bottleneck in tree-based speculative decoding: the trade-off between drafting speed and tree quality. By enabling multiple drafters to collaboratively build draft trees, it offers a practical path to faster LLM inference without sacrificing acceptance rates, which is critical for deploying large models in latency-sensitive production environments.
Technical Details
- Multi-drafter grafting framework: TreeGraft deploys drafters of varying computational costs (a weaker/faster drafter and a stronger/slower drafter) to collaboratively construct a shared draft tree, overcoming the limitation of single-drafter approaches
- Stronger drafter rescoring: The stronger drafter updates scores assigned by the weaker drafter, reselects optimal grafting positions within the tree, and recovers promising paths that were previously abandoned
- Non-destructive expansion integration: Expansions from the stronger drafter are integrated without destroying existing branches, preserving candidates that may still be accepted by the target model during verification
- Lightweight cost-aware scheduler: A scheduler distilled from an offline value system dynamically determines when to call the stronger drafter, balancing drafting quality against computational overhead
- Empirical evaluation: Tested across 10 model pairs and 6 benchmarks, achieving a 15.1% average improvement over the better single-drafter baseline, with peak gains of 26.6%
Industry Insight
- The multi-drafter paradigm presented by TreeGraft suggests a new direction for optimizing speculative decoding systems, where heterogeneous model components can be orchestrated rather than relying on a single fixed drafter
- The use of a distilled scheduler for dynamic drafter selection highlights the growing trend of lightweight meta-decisions that adaptively manage computational resources during inference
- As tree-based speculative decoding matures, frameworks like TreeGraft could become standard components in inference optimization stacks, particularly for organizations deploying large language models at scale where latency and throughput are critical
Disclaimer: The above content is generated by AI and is for reference only.