Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning
When a single policy is trained in parallel across multiple environments (procedural levels, randomized dynamics, curricula), using one shared critic causes value mismatch because different environments assign different expected returns to the same observable state This mismatch systematically shifts sampled advantages within individual environments, reinforcing unhelpful actions while attenuating or reversing useful ones A minimal intervention—providing only a logged environment index to the cr
Analysis
TL;DR
- When a single policy is trained in parallel across multiple environments (procedural levels, randomized dynamics, curricula), using one shared critic causes value mismatch because different environments assign different expected returns to the same observable state
- This mismatch systematically shifts sampled advantages within individual environments, reinforcing unhelpful actions while attenuating or reversing useful ones
- A minimal intervention—providing only a logged environment index to the critic—allows it to separate value targets and resolve the mismatch
- Theoretical analysis using bandit models shows that while oracle processes (no baseline, shared value, environment-specific value) converge to the same optimal policy, their realized learning paths differ sharply
- Multihead conditional critics improve aggregate normalized return by 40.8% across all 16 Procgen games on 600 unseen levels per game, with gains on CartPole, MuJoCo, BipedalWalker, and Procgen
Why It Matters
This work identifies a previously underappreciated mechanism—value mismatch—through which critic sharing degrades stochastic learning dynamics in parallel reinforcement learning, going beyond what scalar estimator variance alone can explain. For AI practitioners running large-scale distributed RL (e.g., RLHF, procedural generation, curriculum learning), this finding suggests that a trivial architectural change can yield substantial performance gains without additional computational cost. It challenges the common assumption that sharing critics is a harmless optimization.
Technical Details
- Problem formulation: Parallel RL training where a single policy interacts with multiple environments simultaneously (e.g., procedurally generated levels, randomized dynamics, curricula), but a single shared critic estimates values across all environments
- Theoretical analysis: Uses illustrative bandit models with multiple environments and a common optimal arm to characterize how value mismatch redistributes sampled policy updates, showing that shared critics reconcile distinct value targets by systematically shifting advantages
- Intervention: A multihead conditional critic that receives only a logged environment index, enabling it to maintain separate value estimates per environment while sharing the actor
- Empirical validation: Controlled experiments on CartPole and MuJoCo demonstrate predicted shifted values, advantages, and performance gaps; more complex tasks (BipedalWalker, Procgen) show more stable learning and higher returns
- Key result: Across 16 Procgen games, the multihead conditional critic improves aggregate normalized return on 600 unseen levels per game by 40.8%
Industry Insight
- Practitioners using distributed RL frameworks (e.g., RLlib, CleanRL, or custom multi-environment setups) should audit whether their critic architecture conditions on environment identity, as unaddressed value mismatch may be silently degrading sample efficiency
- The minimal intervention (environment index conditioning) requires negligible additional parameters and computation, making it a high-ROI change worth adopting in any parallel RL pipeline, especially for procedural generation and curriculum-based training
- This work suggests that future RL benchmarks and training protocols should report per-environment value estimates alongside aggregate metrics to surface value mismatch issues early in development
Disclaimer: The above content is generated by AI and is for reference only.