Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning
Deep reinforcement learning agents using frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations without any sparsity-inducing objective. The first fully-connected layer (FC1) compresses task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong and 5-11 for stochastic Pong, while trainable CNNs activate 55-64 neurons under matched conditions. FC1 sparsity scales with task complexity: 1-11 for Pong,
Analysis
TL;DR
- Deep reinforcement learning agents using frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations without any sparsity-inducing objective.
- The first fully-connected layer (FC1) compresses task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong and 5-11 for stochastic Pong, while trainable CNNs activate 55-64 neurons under matched conditions.
- FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and ~42 for Space Invaders, reflecting task structure rather than a fixed capacity fraction.
- Within-game scaling emerges in identical Pong seeds, producing different numbers of active neurons (5, 7, and 11), which affects achievable performance, suggesting the random projection's usable dimensionality bounds performance.
- Ablation studies confirm the necessity of these active neurons, as removing them crashes performance across two PPO implementations and four games.
- The information bottleneck commits early, with the active set locking by 15-30M steps, while reward turns positive 35-105M steps later.
- Frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio ~10-14), while trainable agents use 51 (participation ratio ~3.6).
Why It Matters
This phenomenon highlights how deep reinforcement learning agents can naturally exploit sparsity in their representations when using frozen, randomly initialized feature extractors, which could lead to more efficient and interpretable models. Understanding this emergent sparsity can inform the design of more effective and resource-efficient reinforcement learning algorithms, particularly in scenarios where computational resources are limited or where interpretability is crucial.
Technical Details
- Sparsity in Fully-Connected Layers: Agents trained with frozen, randomly initialized CNN feature extractors develop extremely sparse representations in the first fully-connected layer (FC1), with only 1-3 neurons active for deterministic Pong and 5-11 for stochastic Pong.
- Task Complexity Scaling: The degree of sparsity scales with task complexity, as evidenced by the number of active neurons increasing from 1-11 for Pong to 19-26 for Breakout and ~42 for Space Invaders.
- Within-Game Variability: Different seeds within the same game produce varying levels of sparsity, affecting performance. For example, one seed with 5 active neurons plateaus at +14 reward, while others with 7 and 11 active neurons reach expert performance (+18.4 and +18.7).
- Necessity of Active Neurons: Ablation studies show that removing these active neurons significantly degrades performance, confirming their critical role in the agent's decision-making process.
- Early Information Bottleneck: The active set of neurons locks in early during training (15-30M steps), well before the reward becomes positive (35-105M steps), indicating an early commitment to a specific representation.
- Comparison with Trainable CNNs: Frozen and trainable CNNs achieve similar rewards but through different structural bottlenecks, with frozen agents using fewer active neurons and higher participation ratios compared to trainable agents.
Industry Insight
- Efficient Model Design: The discovery of emergent sparsity suggests that using frozen, randomly initialized feature extractors can lead to more efficient and compact models, potentially reducing computational costs and improving interpretability.
- Performance Limitations: The variability in sparsity across different seeds indicates that the choice of initial random projections can significantly impact performance, highlighting the need for careful initialization strategies in reinforcement learning.
- Early Commitment to Representations: The early locking of the active neuron set implies that reinforcement learning agents may benefit from early intervention techniques to optimize their representations, potentially leading to faster convergence and better final performance.
Disclaimer: The above content is generated by AI and is for reference only.