Mitigating Exploration Bias in RL for Multi-Instruction Following
RL training for multi-instruction LLMs suffers from exploration bias toward easy instructions due to low initial capability on hard instructions and uniform cumulative reward treatment Two metrics are proposed to quantify exploration bias in instruction following, showing high correlation with model performance A two-stage framework is introduced: Behavioral Bootstrapping (rejection sampling fine-tuning before RL) and Scarcity-Aware Rewards (reward scaling based on instruction difficulty) The pr
Analysis
TL;DR
- RL training for multi-instruction LLMs suffers from exploration bias toward easy instructions due to low initial capability on hard instructions and uniform cumulative reward treatment
- Two metrics are proposed to quantify exploration bias in instruction following, showing high correlation with model performance
- A two-stage framework is introduced: Behavioral Bootstrapping (rejection sampling fine-tuning before RL) and Scarcity-Aware Rewards (reward scaling based on instruction difficulty)
- The proposed methods significantly outperform baselines across three verifiable instruction following benchmarks
- Code has been released publicly
Why It Matters
This work addresses a critical bottleneck in RL-based LLM training where models systematically neglect hard instructions, limiting their real-world utility. For practitioners building instruction-following systems, understanding and mitigating this bias is essential for achieving robust multi-task performance rather than optimizing only for easy cases.
Technical Details
- Exploration Bias Diagnosis: The paper identifies two root causes: (1) the policy model's initial inability to satisfy hard instructions prevents successful exploration signals during RL, and (2) canonical cumulative reward functions treat all instructions equally, creating a greedy incentive to fulfill easy ones
- Behavioral Bootstrapping: A lightweight pre-RL stage using rejection sampling fine-tuning to activate the model's capability on hard instructions before reinforcement learning begins
- Scarcity-Aware Rewards: A novel reward function that assigns higher rewards to rarer/harder instructions based on empirical scarcity rather than treating all fulfilled instructions equally
- Evaluation: Tested across three verifiable instruction following benchmarks, with the proposed metrics showing high correlation with downstream model performance
Industry Insight
- RL training pipelines for instruction following should incorporate difficulty-aware reward shaping rather than relying on naive cumulative rewards, especially for multi-instruction prompts
- Pre-RL bootstrapping via rejection sampling is a cost-effective strategy to unlock hard instruction capability without expensive pre-training modifications
- The proposed bias metrics could serve as early-warning diagnostics for RL training runs, enabling practitioners to detect and correct exploration imbalance before significant compute is wasted
Disclaimer: The above content is generated by AI and is for reference only.