UK's AI Security Institute finds standard benchmarks systematically underestimate what AI agents can actually do
Standard AI benchmarks systematically underestimate agent capabilities because they use fixed, low compute budgets that cap performance before models reach their potential. Increasing test-time compute (token budgets) significantly boosts success rates, with gains up to 25% in software engineering and notable improvements in cybersecurity tasks. There is a strong power-law correlation between the time a human expert takes to complete a task and the number of tokens an AI agent requires. Newer fr
Analysis
TL;DR
- Standard AI benchmarks systematically underestimate agent capabilities because they use fixed, low compute budgets that cap performance before models reach their potential.
- Increasing test-time compute (token budgets) significantly boosts success rates, with gains up to 25% in software engineering and notable improvements in cybersecurity tasks.
- There is a strong power-law correlation between the time a human expert takes to complete a task and the number of tokens an AI agent requires.
- Newer frontier models benefit disproportionately from increased compute compared to older models, showing greater reach, reliability, and efficiency.
- Evaluating AI safety and capability requires measuring performance curves across varying compute budgets rather than relying on single-point fixed-score metrics.
Why It Matters
This research challenges the validity of current AI evaluation standards, suggesting that many reported "limits" of AI agents are artificial constraints imposed by insufficient testing resources rather than inherent model deficiencies. For security researchers and policymakers, this implies that the actual risk posed by autonomous AI agents may be higher than currently assessed, as these systems can achieve significantly more complex outcomes when allowed greater computational effort. Consequently, risk assessments and deployment decisions based on low-budget benchmarks may be dangerously optimistic or misleading.
Technical Details
- Benchmark Variations: The UK's AI Security Institute (AISI) tested frontier models across seven benchmarks, including cybersecurity (METR, AISI), software engineering (TerminalBench 2.0, SWE-Bench Pro), math/academics (Humanity's Last Exam), and health (HealthBench), using token budgets ranging from 1 million to over 100 million.
- Compute Scaling Effects: Success rates increased by approximately 25% in software engineering when budgets rose from 1 million to 10 million tokens. In cybersecurity, about 8% of tasks were only solvable with budgets exceeding 10 million tokens, with some requiring up to 50 million.
- Human-AI Correlation: Analysis of 211 software engineering and 78 cyber tasks revealed a power-law relationship where agent token consumption scales directly with human expert task duration (e.g., a one-week human task correlates with billions of tokens for the agent).
- Model Generational Improvements: Newer models demonstrated superior scaling properties, shifting capability curves along three axes: reach (solving harder tasks), reliability (higher success rates), and efficiency (fewer tokens per task). The time horizon for frontier models doubled every 40–50 days at high budgets, compared to every 4.7 months at low budgets.
- Domain-Specific Plateaus: While domains allowing self-verification (code execution, exploit testing) showed continuous improvement with compute, domains with delayed or missing feedback (like HealthBench) plateaued quickly within standard budgets.
Industry Insight
- Re-evaluate Evaluation Protocols: Organizations must move beyond static benchmark scores and adopt "minimum informative budgets" that test whether model performance continues to scale with compute. Relying on fixed, low-cost evaluations provides an incomplete and potentially unsafe picture of model capabilities.
- Adjust Risk Models for Autonomous Agents: Given that AI agents can achieve significantly higher success rates with increased compute, risk frameworks for deploying autonomous agents in critical sectors (cybersecurity, finance, healthcare) must account for the possibility of "compute-amplified" capabilities that exceed current threat models.
- Monitor Compute Cost Trends: As token costs decrease, high-compute evaluations and attacks become more economically viable. Stakeholders should anticipate that capabilities previously deemed "too expensive" to test or exploit will become accessible, necessitating proactive safety measures that assume higher effective intelligence levels.
Disclaimer: The above content is generated by AI and is for reference only.