Benchmarks disagree on GPT-6 Astra, but its human-beating efficiency on ARC-AGI-3 pulls Chollet's AGI forecast forward
GPT-6 Astra shows contradictory benchmark results: Epoch AI ranks it first overall (169 points across 50+ benchmarks), while Artificial Analysis rates it equal to its predecessor at 61 points, behind Claude Fable 5.1 at 66. The most significant breakthrough is on ARC-AGI-3, where Astra achieves 62.7% on unfamiliar game worlds—far surpassing Sol's 7.8% and Opus 5's 30.2%—and operates more efficiently than the average human solver. Astra is substantially more expensive per token than its predecess
Analysis
TL;DR
- GPT-6 Astra shows contradictory benchmark results: Epoch AI ranks it first overall (169 points across 50+ benchmarks), while Artificial Analysis rates it equal to its predecessor at 61 points, behind Claude Fable 5.1 at 66.
- The most significant breakthrough is on ARC-AGI-3, where Astra achieves 62.7% on unfamiliar game worlds—far surpassing Sol's 7.8% and Opus 5's 30.2%—and operates more efficiently than the average human solver.
- Astra is substantially more expensive per token than its predecessor (2.5x the cost), yet uses only a third of Sol's compute steps and a fifth of Opus 5's, making it cost-competitive on coding tasks versus Anthropic models.
- The model develops its own symbolic shorthand notation for tracking game states and rules, demonstrating on-the-fly symbolic world modeling previously only seen with external harnesses.
- ARC Prize's François Chollet calls the progress "2x faster" than expected and revises his AGI forecast to "sooner," while clarifying that ARC-AGI-3 success is not proof of AGI.
Why It Matters
This article highlights the growing fragmentation in AI benchmarking, where different evaluation frameworks produce contradictory rankings for the same model—critical context for practitioners trying to assess real capability. The ARC-AGI-3 results represent a qualitative shift in how models approach unknown environments, with Astra internalizing symbolic reasoning previously dependent on external tooling, which has implications for how we define and measure general intelligence. The cost-efficiency dynamics also signal a strategic pivot in the frontier model race: raw benchmark scores matter less than the ability to solve complex tasks with fewer compute steps.
Technical Details
- Benchmark divergence: Epoch AI aggregates 50+ benchmarks into a single ECI score (Astra: 169, Sol: 162, Fable 5.1: 163), while Artificial Analysis uses a narrower index covering knowledge, coding, and text comprehension (Astra: 61, Fable 5.1: 66). Astra leads on math, knowledge, and puzzles; Fable 5.1 dominates nearly all coding tests.
- ARC-AGI-3 performance: Astra scores 62.7% on unfamiliar game worlds at ~$26,000 cost, compared to Sol's 7.78% and Opus 5's 30.16%. With OpenAI's proprietary harness (which maintains reasoning chains and summarizes long runs), performance reaches 99.9%, though ARC Prize considers only the standard harness result fair for cross-vendor comparison.
- Reasoning effort paradox: Higher reasoning levels reduce cost on ARC-AGI-3—from $49,791 (no reasoning) to $26,098 (max reasoning)—because Astra solves games in fewer moves, requiring fewer model calls and tokens. The "low" reasoning level (17.5%) underperforms even the no-reasoning baseline (35.2%).
- Symbolic self-modeling: Astra invents an algebra-like DSL to record objects, coordinates, rules, and plans (e.g., "extend8 to3; retract10 to2", "Turn 5: P=(24,20), empty, facing west"). In the PRO-LONG sandbox environment, it writes custom program libraries including board parsers, state models, search algorithms, and planners.
- Efficiency vs. human baseline: Astra cleared 96% of ARC-AGI-3 levels in fewer moves than the median human solver. Human testers earned ~$12.78 per game including compensation; modeling brain metabolism as electricity yields ~0.067 cents per game.
- Other benchmark changes: Hallucination rate on AA-Omniscience drops from 92% to 51%. FrontierMath Erdős: Astra solved 2 of 68 open problems with Lean-verified proofs at $300 per attempt. Astra loses ~80 Elo on GDPval-AA v2 and slips on banking support, SciCode, and long-context reasoning.
Industry Insight
- Benchmarking is becoming unreliable as a sole capability indicator: The stark contradiction between Epoch AI and Artificial Analysis rankings—driven by different test compositions and reasoning-level choices—means practitioners must scrutinize methodology before drawing conclusions about model superiority. The single coding score for Astra at medium reasoning further undermines direct comparisons.
- Symbolic reasoning is migrating from harnesses to base models: Chollet's observation that "harness capabilities are increasingly shifting into the model itself" signals a structural change in model architecture. Future benchmarks will need to account for models that internally replicate what previously required external tooling, and the PRO-LONG sandbox results demonstrate that models can autonomously build complex toolchains when given the opportunity.
- The AGI timeline is compressing, but definitions remain contested: Chollet's revised forecast ("sooner") and the ARC Prize team's insistence that ARC-AGI-3 is "not proof of AGI" reflect a growing tension between rapid empirical progress and conservative philosophical standards. ARC-AGI-4 (expected Q1 2027) will target recursive self-improvement and open innovation, suggesting the benchmarking frontier is moving toward capabilities that are harder to measure but more indicative of general intelligence.
Disclaimer: The above content is generated by AI and is for reference only.