When Does Reward Teach State? A Hidden-Automaton Instrument and the Group-Language Boundary
Introduces a white-box instrument using hidden Deterministic Finite Automata (DFA) to definitively distinguish between true latent state representation and reward-correlated shortcuts in RL agents. Demonstrates that optimizer strength significantly impacts state recovery, with PPO+GAE achieving partial recovery despite high seed variance, whereas weaker optimizers fail completely. Identifies permutation (group-language) structure as a pre-training warning sign for "perception gaps," flagging tas
Analysis
TL;DR
- Introduces a white-box instrument using hidden Deterministic Finite Automata (DFA) to definitively distinguish between true latent state representation and reward-correlated shortcuts in RL agents.
- Demonstrates that optimizer strength significantly impacts state recovery, with PPO+GAE achieving partial recovery despite high seed variance, whereas weaker optimizers fail completely.
- Identifies permutation (group-language) structure as a pre-training warning sign for "perception gaps," flagging tasks where latent states are not linearly recoverable with 86% precision.
- Establishes a critical distinction between perception gaps (state exists but isn't linearly recoverable) and planning gaps (state is recoverable but unused), proving high reward does not equal task understanding.
Why It Matters
This research provides a rigorous methodological framework for auditing RL agents, moving beyond behavioral metrics to inspect internal representations against ground-truth latent states. For practitioners, it highlights that standard reward maximization is insufficient for verifying model comprehension, necessitating new evaluation protocols that account for structural properties of the task environment.
Technical Details
- White-Box Instrument: Tasks are modeled as hidden DFAs where the agent observes symbol streams and makes intermittent choices, receiving a single sparse terminal reward upon acceptance. This setup provides known optimal returns and exact latent states for probing.
- Optimizer Analysis: Comparative testing shows that weak on-policy RL fails to learn latent states (performance at chance), while PPO+GAE recovers states partially but exhibits high variance across random seeds.
- Structural Warning Signs: The study analyzes task structure, specifically identifying permutation/group-language structures in the transition function as predictors of perception gaps, validated on 153 capacity-controlled automata.
- Auxiliary Learning: A label-free auxiliary task was tested, showing it remains vacuous when observations lack state information and scales in utility proportional to observation informativeness.
Industry Insight
- Audit Protocols: Developers should implement structural analysis of task environments before training to predict potential perception gaps, rather than relying solely on post-hoc reward evaluation.
- Evaluation Metrics: Shift from reward-centric metrics to include latent state recoverability tests, particularly for safety-critical applications where shortcut learning could lead to catastrophic failures.
- Algorithm Selection: Be cautious with PPO in sparse reward settings due to high seed variance; consider stronger optimizers or auxiliary tasks if latent state fidelity is required for downstream planning.
Disclaimer: The above content is generated by AI and is for reference only.