When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic
The learned termination rule in Option-Critic is functionally useless: when the termination test and option-picking policy observe identical values, it fires at every step, effectively collapsing to always-terminate behavior The authors introduce "policy necrosis," a phenomenon where sub-policies inside options barely explore and permanently lock onto their first seemingly-good action, affecting approximately 60% of states in a typical option Adding more options does not improve any individual o
Analysis
TL;DR
- The learned termination rule in Option-Critic is functionally useless: when the termination test and option-picking policy observe identical values, it fires at every step, effectively collapsing to always-terminate behavior
- The authors introduce "policy necrosis," a phenomenon where sub-policies inside options barely explore and permanently lock onto their first seemingly-good action, affecting approximately 60% of states in a typical option
- Adding more options does not improve any individual option's performance; instead, it reduces the joint probability of all options failing in the same state from 59% to 4%, which drives the observed performance gains
- Forcing termination at every step preserves the option-count performance curve, confirming the termination mechanism contributes nothing meaningful
- Restoring exploration within options repairs necrotic states, enabling a single option to solve the task
Why It Matters
This paper provides a critical deconstruction of Option-Critic, one of the foundational architectures for hierarchical reinforcement learning, revealing that its celebrated performance improvements from adding options stem from redundant coverage rather than genuine sub-policy learning. For practitioners building or extending hierarchical RL systems, these findings challenge assumptions about the value of option count and highlight exploration as the true bottleneck.
Technical Details
- Termination rule analysis: The paper proves that when the termination test and the option-selection policy read the same value function, the termination condition fires at every timestep, making the learned rule equivalent to always terminating. In cases where the policy explores but the termination test does not (as in standard Option-Critic), the termination rule can actively block exploration, suffering Omega(T) regret compared to O(log T) for always-terminating.
- Policy necrosis: The authors define and detect a state-level phenomenon where the policy inside an option ceases exploration almost entirely, locking onto the first action with favorable initial estimates. They quantify this by finding three-fifths of states exhibit necrotic behavior in a typical option.
- Redundant coverage mechanism: Performance scaling with option count is explained through a probabilistic lens: additional options do not make any single option better, but they reduce the probability that all options fail simultaneously in a given state, from 59% down to 4%.
- Empirical validation: The authors demonstrate that restoring exploration within options repairs necrotic states, allowing even a single option to solve tasks that previously required multiple options.
Industry Insight
- Hierarchical RL practitioners should prioritize exploration mechanisms within sub-policies over simply increasing option count; the marginal gains from additional options appear to come from coverage redundancy rather than genuine skill diversification.
- The policy necrosis phenomenon likely generalizes beyond Option-Critic to other hierarchical RL frameworks that use fixed-duration or internally-terminated sub-policies, suggesting a need for built-in exploration guarantees at the sub-policy level.
- The finding that termination rules can actively harm exploration warrants re-examination of termination learning objectives in existing hierarchical RL systems, particularly in safety-critical applications where option switching behavior directly impacts system reliability.
Disclaimer: The above content is generated by AI and is for reference only.