Beyond the price per token: Choosing the right OpenAI model on Amazon Bedrock for your workload
Organizations should evaluate AI models by cost per correct outcome rather than cost per token, as production workloads buy results, not tokens OpenAI's GPT-5.6 Luna on Amazon Bedrock achieved the lowest observed cost per correct answer ($0.0021 on AIME) after an 80% price reduction, outperforming even cheaper-per-token models like nano Agent trajectory costs are dominated by turn count due to quadratic growth in billed input tokens as conversation context accumulates across turns The benchmarki
Analysis
TL;DR
- Organizations should evaluate AI models by cost per correct outcome rather than cost per token, as production workloads buy results, not tokens
- OpenAI's GPT-5.6 Luna on Amazon Bedrock achieved the lowest observed cost per correct answer ($0.0021 on AIME) after an 80% price reduction, outperforming even cheaper-per-token models like nano
- Agent trajectory costs are dominated by turn count due to quadratic growth in billed input tokens as conversation context accumulates across turns
- The benchmarking harness (openai-on-aws/benchmarks-openai) evaluates models across three dimensions: single-call accuracy, multi-turn agent trajectories, and professional deliverable quality
- Capability tiers are clearly visible: Sol solves 75% of AIME problems versus 37% for mini, with similar gaps on GPQA Diamond (68% vs 43%) and MMLU-Pro (82% vs 59%)
Why It Matters
This article provides a practical framework for AI practitioners to move beyond simplistic token-based pricing comparisons and evaluate models based on real-world outcome costs, which is critical for production deployments where accuracy, token efficiency, and agent turn counts compound to determine true expenditure. The open-source benchmarking harness enables organizations to reproduce results on their own workloads before making model selection decisions, reducing the risk of costly misallocations.
Technical Details
- The benchmarking harness evaluates five models (gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol on Amazon Bedrock; gpt-5.4-mini and gpt-5.4-nano on OpenAI API) through a single identical code path using the OpenAI Responses API, ensuring consistent evaluation logic
- Three benchmark categories were used: single-call accuracy on AIME (mathematics), GPQA Diamond (graduate-level science), and MMLU-Pro; multi-turn agent trajectories on DeepSearchQA with live web_search and fetch_page tools; and rubric-graded professional deliverables scored via deterministic checks plus an LLM judge (gpt-5.5)
- Cost per correct answer was calculated by dividing total spend across all attempts by the number of correct answers, revealing that Luna's token efficiency and reduced billing (with reasoning disabled) made it 25% cheaper per correct answer even before the July 30, 2026 price reduction
- Agent trajectory measurement used client-managed history with store:false, causing every turn to re-send the full conversation context, resulting in approximately quadratic growth in cumulative billed input tokens relative to turn count
- Sample sizes ranged from 48–198 items, with results written as timestamped JSON files and frozen prompt hashes recorded for reproducibility
Industry Insight
- Model selection should be driven by outcome economics rather than sticker price; organizations running cost-optimized models like mini or nano should evaluate whether upgrading to newer Bedrock models delivers sufficient accuracy and token efficiency gains to justify the switch
- For agentic workloads, minimizing turn count is as critical as minimizing per-turn cost, since context accumulation creates quadratic billing growth—architecting agents for fewer, more decisive turns can dramatically reduce total expenditure
- The open-source benchmarking approach should be adopted as a standard practice, with teams running the harness against their own task distributions before committing to model procurement decisions, as small performance gaps may be directional rather than statistically significant
Disclaimer: The above content is generated by AI and is for reference only.