BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?
RLVR improves LLM reasoning performance but the mechanism is debated: does it expand reasoning capability or merely improve sampling efficiency? The authors introduce BODHI-Trees, a semantic-equivalence-based tree structure extracted from mathematical reasoning traces, to distinguish stylistic entropy from genuine inferential branching Policy entropy collapse in RLVR models is semantic, not just syntactic, with significant reduction in semantic branching entropy RLVR enhances constraint adherenc
Analysis
TL;DR
- RLVR improves LLM reasoning performance but the mechanism is debated: does it expand reasoning capability or merely improve sampling efficiency?
- The authors introduce BODHI-Trees, a semantic-equivalence-based tree structure extracted from mathematical reasoning traces, to distinguish stylistic entropy from genuine inferential branching
- Policy entropy collapse in RLVR models is semantic, not just syntactic, with significant reduction in semantic branching entropy
- RLVR enhances constraint adherence and backtracking but constricts the space of continuations, trading rollout diversity for sample efficiency
Why It Matters
This paper directly addresses a fundamental question in the RLVR training paradigm: whether performance gains reflect genuine reasoning expansion or optimized sampling. For AI practitioners, understanding this distinction is critical for deciding when to invest in RLVR versus other training approaches, and for interpreting benchmark improvements honestly.
Technical Details
- BODHI-Trees: A novel tree structure extracted from mathematical reasoning traces based on semantic equivalence, enabling quantification of inferential branching versus stylistic variation
- Controlled maze-solving experiments: Used as a testbed to isolate and measure test-time exploration behavior in RLVR-trained LLMs
- Entropy decomposition: The framework separates policy entropy into syntactic/stylistic components and genuine semantic branching entropy, revealing that RLVR collapses both
- Key finding: RLVR-trained models show reduced semantic branching entropy, indicating constricted continuation spaces despite improved constraint adherence and backtracking
Industry Insight
- RLVR should be viewed as a sample-efficiency optimization rather than a reasoning-capability expansion; practitioners should combine it with methods that preserve exploratory diversity for complex, open-ended tasks
- The BODHI framework offers a diagnostic tool for evaluating whether future RLVR variants maintain semantic diversity, which could become a standard metric alongside accuracy benchmarks
- Over-reliance on RLVR without diversity-preserving mechanisms may lead to models that perform well on constrained reasoning tasks but struggle with genuinely novel inference paths.
Disclaimer: The above content is generated by AI and is for reference only.