Custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge
Amazon Nova Forge introduces Bring Your Own Orchestration (BYOO) for multi-turn reinforcement learning, allowing custom reward functions to run in user-managed environments via Amazon ECS Multi-turn RFT optimizes cumulative reward across entire agent trajectories (tool calls, code execution, error recovery) rather than grading single responses GRPO ranks K model rollouts per conversation and updates weights based on normalized reward advantages within each group Reward signals only influence lea
Analysis
TL;DR
- Amazon Nova Forge introduces Bring Your Own Orchestration (BYOO) for multi-turn reinforcement learning, allowing custom reward functions to run in user-managed environments via Amazon ECS
- Multi-turn RFT optimizes cumulative reward across entire agent trajectories (tool calls, code execution, error recovery) rather than grading single responses
- GRPO ranks K model rollouts per conversation and updates weights based on normalized reward advantages within each group
- Reward signals only influence learning through within-group variation; constant-value components contribute zero gradient despite appearing healthy in training curves
- A real-world pitfall was identified where the highest-weighted reward component silently contributed no learning signal due to zero variance across completions
Why It Matters
This is highly relevant for AI practitioners building agentic systems, as multi-turn RL represents a critical shift from single-turn chat models to autonomous agents that can execute tools, write code, and recover from errors. The emphasis on reward function design—particularly the danger of silent signal collapse—provides practical guidance that could prevent costly training failures. The comparison showing RL improving out-of-distribution generalization while SFT degrades it is a compelling argument for adopting RFT approaches.
Technical Details
- BYOO Architecture: Multi-turn rollouts exceed Lambda's 15-minute invocation limit, so Nova Forge delegates to user-managed containers on Amazon ECS. The environment manages user simulators, code execution, and verifiers, returning
aggregate_reward_scoreand optionalmetrics_listper sample. - GRPO Algorithm: For each conversation, GRPO samples K rollouts, ranks them using the custom reward function, and updates model weights based on normalized advantages within the batch. The reward signal only matters through within-group variation.
- Reward Function Types: Supports rule-based verifiable rewards (checking output correctness) and LLM-as-Judge approaches. The grader is written in code, not a separately trained reward model.
- Infrastructure Stack: Requires Amazon SageMaker HyperPod cluster on ECS, S3 bucket for rollout data and checkpoints, and the Nova Customization SDK. Configuration set via
cdk.jsonwithuse_custom_envandcustom_env_idflags. - OOD Performance: Cites Chu et al. 2025 showing RL improves out-of-distribution generalization across all task variants while SFT degrades performance under equal-compute post-training from shared checkpoints.
Industry Insight
- Reward engineering is the bottleneck: The article's central warning—that subtly wrong rewards teach wrong behaviors while training curves look healthy—should drive investment in reward instrumentation and component-level monitoring before scaling multi-turn RL projects.
- BYOO vs serverless tradeoff: Teams should evaluate whether they need the flexibility of BYOO (custom environments, code execution, complex verifiers) or can use the serverless multi-turn RL option for simpler use cases, as infrastructure complexity directly impacts iteration speed.
- RL over SFT for agentic tasks: The OOD generalization results suggest that for production agent deployments where tasks vary beyond training distribution, RFT with GRPO should be preferred over SFT, despite the additional complexity of reward design.
Disclaimer: The above content is generated by AI and is for reference only.