Beyond Next-Token Prediction: An RLVR Proof of Concept for Tool-Use Agents on Atlassian Workflows 超越下一个Token预测：Atlassian工作流中工具使用代理的RLVR概念验证

TL;DR

• The study addresses the objective mismatch between next-token prediction and precise API execution in enterprise SaaS workflows, which often leads to silent failures like dropped fields or hallucinated tools.
• Researchers developed five synthetic environments mimicking Jira REST v3 and Confluence v2 APIs to test Reinforcement Learning with Verifiable Rewards (RLVR), computing rewards directly from tool-call traces without live APIs or human labeling.
• RL-trained policies significantly outperformed prompted baselines, lifting average rewards from a range of 0.35–0.92 to 0.95–1.00 across four non-degenerate scenarios, with Confluence page creation showing the largest improvement (0.35 to 1.00).
• The approach demonstrates the viability of outcome-optimized small models (Qwen3-1.7B and Qwen3.5-4B) for niche enterprise APIs, though it highlights limitations in scalability due to the manual effort required for crafting verifiable rewards.

Why It Matters

This research provides a concrete pathway for deploying LLMs in critical enterprise environments where precision is paramount, moving beyond generative capabilities to reliable action execution. By validating RLVR on schema-fidelity emulations, it offers practitioners a method to reduce silent failures in API interactions without relying on costly human-in-the-loop verification or unstable learned judges.

Technical Details

• Methodology: Utilized Reinforcement Learning with Verifiable Rewards (RLVR) combined with GRPO training, applying rewards directly based on the correctness of tool-call traces against API schemas.
• Environment: Created five synthetic environments emulating Jira REST v3 and Confluence v2 APIs, ensuring schema fidelity while eliminating the need for live API calls or external judges.
• Models Evaluated: Tested Qwen3-1.7B and Qwen3.5-4B models, comparing RL-trained policies against prompted baselines.
• Performance Metrics: Achieved near-perfect scores (0.95–1.00) in four scenarios, with specific gains such as increasing Confluence page creation success from 0.35 to 1.00.
• Limitations Identified: Hand-crafting verifiable rewards does not scale easily beyond a few endpoints, and some scenarios (like ticket-transition) showed saturating reward shapes where baseline models already performed optimally.

Industry Insight

Enterprise AI solutions must prioritize verifiable, deterministic reward structures over probabilistic judgment models to ensure reliability in mission-critical workflows like IT service management. Developers should invest in high-fidelity synthetic testing environments that mirror production API schemas to train agents effectively before deployment. Future work needs to address the scalability of reward engineering, potentially through automated schema-to-reward translation, to make RLVR applicable to broader API ecosystems.

TL;DR

• 针对LLM在特定企业API调用中因“下一词预测”目标与“精确工具调用”需求不匹配导致的静默失败问题，提出使用可验证奖励强化学习（RLVR）进行优化。
• 构建了模拟Jira REST v3和Confluence v2 API的五种合成环境，完全基于工具调用轨迹计算奖励，无需真实API、学习裁判或人工标注。
• 在Qwen3-1.7B和Qwen3.5-4B模型上验证，RL训练策略将四个非退化场景的平均奖励从基线范围（0.35-0.92）提升至0.95-1.00。
• 最大提升出现在Confluence页面创建任务中（从0.35提升至1.00），证明了小模型在垂直领域API调用上的潜力。
• 研究指出局限性：手工设计可验证奖励难以扩展至大量端点，且部分场景（如工单转换）存在奖励饱和现象，基线模型已接近满分。

为什么值得看

本文通过具体的Proof of Concept展示了RLVR技术在解决LLM工具调用精度问题上的有效性，为垂直领域企业级Agent的开发提供了新的技术路径。它揭示了纯监督学习在复杂API交互中的局限，并强调了基于环境反馈的强化学习在提升任务成功率方面的关键作用。

技术解析

• 问题定义：传统LLM训练目标是预测下一个token，而在企业SaaS工作流中，成功标准是正确调用API端点、传递嵌套参数并按顺序执行。这种目标错配导致字段丢失、幻觉工具或过早停止等静默失败。
• 实验环境：构建了五个合成环境，高保真模拟Jira REST v3和Confluence v2 API。奖励函数完全基于工具调用轨迹计算，实现了无真实API调用、无学习裁判模型、无人工标注闭环的自动化评估体系。
• 模型与算法：使用GRPO（Group Relative Policy Optimization）作为训练框架，对Qwen3-1.7B和Qwen3.5-4B两个规模的模型进行评分和训练。
• 性能表现：在四个奖励非退化的场景中，RL训练后的策略显著提升了平均奖励分数。特别是在Confluence页面创建场景中，分数从0.35跃升至1.00，表明模型能够完美掌握该特定API的调用逻辑。

行业启示

• 垂直领域Agent优化方向：对于需要高精度API调用的企业级应用，仅靠指令微调（SFT）可能不足以解决复杂的参数嵌套和顺序问题，引入基于可验证奖励的强化学习（如RLVR）是提升可靠性的有效手段。
• 数据与标注成本考量：虽然RLVR效果显著，但其依赖手工设计的可验证奖励函数，这在端点众多、逻辑复杂的系统中扩展性较差。企业需权衡开发自定义奖励函数的成本与收益，或探索自动化奖励生成的替代方案。
• 小模型潜力挖掘：研究表明，即使是参数量较小的模型（如1.7B-4B），经过针对性的RLVR训练后，也能在特定狭窄任务上达到甚至超越更大模型的基线表现，这为降低企业部署LLM Agent的算力成本提供了新思路。