Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
Replaces standard entropy-based exploration in self-play RL with forward KL divergence toward the network's own MCTS prior, enabling more targeted exploration of promising moves Achieves tactical accuracy improvements (puzzle accuracy: 93.9% → 94.9%, mate-in-four: 77% → 81%) while maintaining searchless playing strength Demonstrates that tactical accuracy and playing strength are dissociable metrics — puzzle-only fine-tuning yields largest tactical gains but costs ~260 Elo Without regularization
Analysis
TL;DR
- Replaces standard entropy-based exploration in self-play RL with forward KL divergence toward the network's own MCTS prior, enabling more targeted exploration of promising moves
- Achieves tactical accuracy improvements (puzzle accuracy: 93.9% → 94.9%, mate-in-four: 77% → 81%) while maintaining searchless playing strength
- Demonstrates that tactical accuracy and playing strength are dissociable metrics — puzzle-only fine-tuning yields largest tactical gains but costs ~260 Elo
- Without regularization, self-play collapses onto a single line of play; the prior-directed approach preserves near-miss solutions that mass-covering priors keep alive
- Forward-KL prior statistically ties with reverse-KL anchor, though the latter concentrates twice as hard and drops harder solutions from its support
Why It Matters
This work addresses a fundamental limitation in distilling search-based chess engines into searchless networks: imitation of search visit counts is an imperfect proxy for actual playing strength. By introducing prior-directed exploration, the authors show how reinforcement learning can be better aligned with the structural knowledge already encoded in a distilled policy, offering a template for improving single-pass decision-making in other domains where search-imitation has been the default approach.
Technical Details
- Architecture: Built on Leela Chess Zero's Chessformer, a searchless network that plays from a single forward pass, distilled from AlphaZero-style MCTS visit counts
- Exploration mechanism: Replaces reverse KL (toward uniform) entropy bonus with forward, mass-covering KL divergence toward the network's own MCTS prior, ensuring exploration concentrates on moves the prior deems promising
- Entropy-adaptive sampling temperature: Dynamically adjusted based on the value head's outcome uncertainty, sharpening once a position is resolved
- Training: Approximately 2,000 self-play RL fine-tuning steps with matched-compute sweeps comparing tactical accuracy against playing strength
- Benchmarks: 100,000-puzzle suite for tactical accuracy; mate-in-four accuracy; Elo rating for playing strength
Industry Insight
- The dissociation between tactical accuracy and playing strength challenges the common assumption that puzzle-solving performance directly translates to stronger play — practitioners should evaluate both metrics independently when assessing model improvements
- Prior-directed exploration offers a principled alternative to generic entropy bonuses in RL systems, particularly valuable when distilling search-based policies into efficient single-pass architectures
- The finding that puzzle-only fine-tuning sacrifices ~260 Elo while yielding the largest tactical gains suggests that domain-specific optimization without strategic grounding can produce misleading performance profiles; balanced regularization is essential for real-world deployment
Disclaimer: The above content is generated by AI and is for reference only.