Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks 基于进度条件的群体策略优化用于长周期代理任务

TL;DR

• The paper proposes Progress-conditioned Group Policy Optimization (ProGPO) to address sampling imbalance in group-based policy optimization for long-horizon agentic tasks.
• ProGPO assigns higher relative advantages to trajectories or steps that visit more new states when all samples in a group receive zero outcome reward, breaking the self-reinforcing credit trap.
• Experiments on ALFWorld and WebShop benchmarks with Qwen2.5-1.5/7B-Instruct show consistent improvements over group-based baselines, especially on hard tasks.

Why It Matters

This work addresses a critical challenge in training LLM agents for complex, long-horizon tasks where sparse rewards often lead to ineffective learning due to sampling bias. By introducing progress-based advantage assignment during failure-dominated rollouts, ProGPO provides a practical solution to escape local optima and improve sample efficiency—directly impacting how researchers design reward structures and optimization strategies for autonomous agents in real-world environments.

Technical Details

• Group-based policy optimization compares trajectories within a group using outcome rewards but suffers from sampling imbalance: repeated low-effect actions dominate high-probability regions while useful state-changing actions are under-sampled.
• When all samples in a group yield zero reward (all-failed groups), ProGPO switches to first-visit observation coverage as a proxy for progress, assigning higher advantages to steps that explore new states.
• This mechanism breaks the "credit trap" where failure-dominated sampling prevents correction of suboptimal policies, enabling continued exploration even without positive outcome signals.
• Evaluated on two challenging benchmarks—ALFWorld (interactive 3D environment) and WebShop (e-commerce task)—using Qwen2.5-1.5 and Qwen2.5-7B Instruct models, demonstrating consistent gains particularly on harder task variants.

Industry Insight

As AI agents increasingly tackle multi-step, real-world problems with sparse feedback, methods like ProGPO offer a scalable way to maintain exploration pressure without dense reward engineering. Practitioners should consider integrating progress-aware advantage functions into existing group-based RL frameworks when dealing with long-horizon tasks, especially in domains where success is rare but incremental state changes indicate meaningful progress. This approach could significantly reduce training time and improve final performance in production-grade agentic systems.

TL;DR

• 针对长时程智能体任务中基于群体的策略优化（Group Policy Optimization）存在的采样不平衡问题，提出Progress-conditioned Group Policy Optimization (ProGPO)方法。
• ProGPO在群体所有样本均获得零奖励时，首次访问观测覆盖率机制被激活，赋予访问更多新状态的轨迹或步骤更高相对优势。
• 实验表明，ProGPO在ALFWorld和WebShop两个具有挑战性的智能体基准测试中，使用Qwen2.5-1.5/7B-Instruct模型时，一致优于群体基线方法，尤其在困难任务上提升显著。

为什么值得看

该研究解决了长时程稀疏奖励场景下智能体训练中的关键信用分配难题，通过创新的进度条件化机制有效打破“失败主导的采样循环”，为构建更鲁棒的自主Agent提供了新的技术路径。对于从事强化学习、多步规划及大模型代理开发的从业者而言，其提出的改进策略具有直接的参考价值和应用潜力。

技术解析

• 核心问题：传统群体策略优化依赖组内轨迹对比来从稀疏结果奖励中学习，但在长时程任务中易出现采样偏差——重复性或低效动作占据高概率区域，而能改变状态的关键动作被低估，导致大量全失败 rollout 组，使奖励信号无法指导策略修正，形成自我强化的信用陷阱。
• 解决方案（ProGPO）：引入“进度条件”机制。仅当一组内所有样本的最终 outcome reward 均为零时，才启用 first-visit observation coverage（首次访问观测覆盖度）作为辅助优势估计依据。即鼓励那些在探索过程中访问了更多新状态的动作或轨迹，因为访问新状态是完成任务的前提。
• 模型与数据集：实验基于 Qwen2.5-1.5-Instruct 和 Qwen2.5-7B-Instruct 两个规模的 LLM 代理，在 ALFWorld（虚拟家居环境）和 WebShop（在线购物模拟）两个 benchmark 上进行评估，这两个环境均以长序列决策和稀疏奖励著称。
• 性能表现：ProGPO 相较于标准的 group-based policy optimization baseline，在两类任务上均表现出稳定且显著的性能提升，特别是在难度较高的子任务或初始阶段成功率极低的情况下增益更为明显。

行业启示

• 面向复杂任务的Agent训练范式升级：随着AI Agent向更长 horizon、更高自由度方向发展，单纯依赖最终结果反馈的训练方式将愈发失效；结合中间过程状态变化（如探索新颖性）进行引导将成为必要补充，ProGPO为此类混合奖励设计提供了可落地的框架。
• 开源生态与大模型协同优化机会凸显：该方法成功应用于Qwen系列模型，证明其具备良好泛化性与工程适配性，未来可与主流RLHF/PPO工具链集成，推动大模型智能体在真实世界应用中的可靠性突破。