I Improved My PPO Agent by 45%. It Still Lost to Simple Rules
Reward redesign improved PPO drone navigation success from 35.1% to 44.2%, primarily by reducing timeout failures, but shifted failure modes toward stuck behavior and collisions Reducing lidar sensor resolution from 16 to 8 rays paradoxically improved performance to 50.8%, while increasing to 32 rays degraded it to 36.0%, suggesting observation complexity can hinder learning under fixed training budgets Progress reward ablation caused success to collapse from 50.8% to 0.9%, revealing extreme dep
Analysis
TL;DR
- Reward redesign improved PPO drone navigation success from 35.1% to 44.2%, primarily by reducing timeout failures, but shifted failure modes toward stuck behavior and collisions
- Reducing lidar sensor resolution from 16 to 8 rays paradoxically improved performance to 50.8%, while increasing to 32 rays degraded it to 36.0%, suggesting observation complexity can hinder learning under fixed training budgets
- Progress reward ablation caused success to collapse from 50.8% to 0.9%, revealing extreme dependence on dense reward shaping rather than robust learned navigation
- Simple hand-written heuristics (84.3% success with wall-avoiding greedy policy) dramatically outperformed the best PPO policy (50.8%), exposing a fundamental performance gap
- The project's core lesson: improving metrics is not the same as solving the task, and strong baselines are essential for honest evaluation
Why It Matters
This case study serves as a critical reality check for AI practitioners investing heavily in reinforcement learning for robotics and navigation tasks, demonstrating that reward engineering and hyperparameter tuning alone cannot guarantee competitive performance. It highlights the often-overlooked importance of establishing strong heuristic baselines before declaring an RL approach successful, preventing premature conclusions about system capabilities.
Technical Details
- Environment: Custom 2D drone-navigation task where the agent receives relative goal position, velocity, heading, and lidar-style ray observations, producing continuous heading and speed commands via PPO
- Reward redesign: Introduced timeout penalties, stall penalties, low-speed penalties, stuck detection, and stricter termination logic to counter passive drifting behavior; best configuration reduced timeouts from 48.2% to 7.2% but increased collisions and stuck episodes
- Sensor ablation: Tested 8, 16, and 32 lidar rays under identical training conditions; 8 rays achieved 50.8% success (best), 16 rays achieved 44.2%, and 32 rays achieved only 36.0%, indicating that higher observation dimensionality increased optimization difficulty without proportional benefit
- Progress reward ablation: Removing the progress reward term caused success to plummet to 0.9% with a 99.1% collision rate, demonstrating that the policy had not learned robust navigation but rather depended entirely on dense shaping signals
- Heuristic baselines: Four hand-written policies were tested—random, greedy goal-following, obstacle-aware, and wall-avoiding greedy—with the latter achieving 84.3% success and only 0.3% collision rate, far exceeding all PPO variants
Industry Insight
- Practitioners should always establish and compare against simple heuristic baselines before investing in complex RL training, as learned policies may appear improved relative to a weak starting point while still being outperformed by straightforward rule-based systems
- Reward shaping requires careful ablation studies; dense shaping signals can create fragile policies that collapse when signals are removed, which is a significant concern for real-world deployment where reward functions may be incomplete or noisy
- Reducing observation complexity can sometimes improve RL performance under fixed computational budgets, suggesting that sensor simplification and observation-space design deserve equal attention alongside reward engineering and architecture choices
Disclaimer: The above content is generated by AI and is for reference only.