When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL
Proposes a principled communication gating mechanism for multi-agent RL using KL divergence between belief distributions instead of learned binary gates via REINFORCE Agents maintain belief distributions over latent world states via softmax over LSTM hidden states, communicating only when disagreement exceeds a fixed threshold On the harder Predator-Prey 20×20 benchmark, KL-belief with ε=0.5 outperforms IC3Net by 1.47 average steps and 11 percentage points in success rate with tighter variance T
Analysis
TL;DR
- Proposes a principled communication gating mechanism for multi-agent RL using KL divergence between belief distributions instead of learned binary gates via REINFORCE
- Agents maintain belief distributions over latent world states via softmax over LSTM hidden states, communicating only when disagreement exceeds a fixed threshold
- On the harder Predator-Prey 20×20 benchmark, KL-belief with ε=0.5 outperforms IC3Net by 1.47 average steps and 11 percentage points in success rate with tighter variance
- The belief head provides two orthogonal benefits: principled gating when beliefs diverge, and improved latent representations that boost coordination even when gating is inactive
- On MPE simple_spread, the approach improves mean reward by 12 points and reduces variance by 26× compared to baselines
Why It Matters
This work addresses a fundamental gap in multi-agent RL—deciding when to communicate—by replacing high-variance, uninterpretable learned gating with a mathematically grounded KL divergence criterion. For practitioners building multi-agent systems, this offers a more stable and interpretable alternative to REINFORCE-based communication gates, with demonstrated gains in both performance and training consistency on standard benchmarks.
Technical Details
- Each agent maintains a belief distribution over a latent world state, computed as a softmax over its LSTM hidden state; communication is triggered when the KL divergence between co-agents' belief distributions exceeds a fixed threshold ε
- The approach is evaluated on the Predator-Prey benchmark from IC3Net (10×10 and 20×20 grids) with 5 seeds each, and on the MPE simple_spread environment, comparing against IC3Net, CommNet, and an independent controller
- Threshold ablation over ε ∈ {0.1, 0.3, 0.5, 1.0} reveals an inverted U-shape performance curve, with ε=0.5 optimal on PP 20×20 (73.84 avg steps, 42% success vs. IC3Net's 75.31 steps and 31%)
- On MPE, the belief head improves mean reward by 12 points and reduces variance by 26× even when gating is inactive, indicating the representation learning benefit is orthogonal to the gating mechanism
- The method eliminates the high-variance policy gradient signal inherent in IC3Net's learned binary gate, replacing it with a deterministic, threshold-based communication decision
Industry Insight
- The inverted U-shape in threshold ablation suggests that optimal communication frequency is non-trivial and environment-dependent; practitioners should treat the KL threshold as a critical hyperparameter requiring careful tuning rather than a fixed constant
- The dual benefit of improved representations plus principled gating implies that adding belief heads to existing multi-agent architectures may yield performance gains even without activating communication gating, making this a low-risk modification
- As multi-agent systems scale to more complex, sparse-communication scenarios, deterministic gating based on belief divergence offers a more deployable alternative to gradient-based gates, which suffer from instability and lack interpretability in production settings
Disclaimer: The above content is generated by AI and is for reference only.