Google updates Android Bench with new LLMs, but Gemini still lags behind
Google has significantly updated the Android Bench leaderboard, introducing a new evaluation framework and adding eight major new models to the test suite. Anthropic's Claude Fable 5 leads the rankings with 84.5% accuracy, outperforming OpenAI’s GPT 5.4 and Google’s own Gemini 3.1 Pro, which ranks fifth. The update introduces critical cost and efficiency metrics, revealing that while top performers have high token costs, some models like Gemini 3.5 Flash suffer from extreme runtime inefficiencie
Analysis
TL;DR
- Google has significantly updated the Android Bench leaderboard, introducing a new evaluation framework and adding eight major new models to the test suite.
- Anthropic's Claude Fable 5 leads the rankings with 84.5% accuracy, outperforming OpenAI’s GPT 5.4 and Google’s own Gemini 3.1 Pro, which ranks fifth.
- The update introduces critical cost and efficiency metrics, revealing that while top performers have high token costs, some models like Gemini 3.5 Flash suffer from extreme runtime inefficiencies.
- Google is transitioning to the Harbor framework to simplify testing and encourage community contributions, allowing developers to submit new tasks and benchmarks.
Why It Matters
This update highlights the intensifying competition among LLM providers in specialized coding domains, demonstrating that proprietary models from competitors like Anthropic and OpenAI currently surpass Google’s Gemini in Android-specific tasks. For AI practitioners, it underscores the importance of evaluating models not just on accuracy, but on operational efficiency, including cost per task and execution time, which are crucial for scalable agentic development workflows.
Technical Details
- Benchmark Scope: Android Bench evaluates LLMs on a suite of 100 Android development tasks, measuring accuracy, cost, and efficiency.
- New Models Evaluated: The leaderboard includes recent heavy-hitters such as Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max.
- Performance Metrics: Claude Fable 5 achieved 84.5% accuracy. Cost analysis shows Fable 5 and GPT 5.5 exceeding $130 for the benchmark, while Gemini 3.1 Pro cost $87. Notably, Gemini 3.5 Flash had the highest cost ($165) due to a 28-hour runtime.
- Framework Shift: Google adopted the Harbor framework as a testing sandbox to standardize evaluations, making it easier for developers to run, evaluate, and share results. Historical data remains archived for comparison.
Industry Insight
- Strategic Implications for Google: As Google pivots toward agentic development, the underperformance of Gemini relative to competitors poses a risk to ecosystem adoption. This may accelerate efforts to improve model capabilities or incentivize developer usage through other means.
- Cost-Efficiency Trade-offs: The data reveals a significant variance in operational costs. Practitioners must balance accuracy with budget constraints; high-accuracy models may be prohibitively expensive for large-scale automated testing, necessitating hybrid strategies.
- Community-Driven Benchmarking: The shift to the Harbor framework signals a move toward decentralized validation. Organizations should monitor community-submitted tasks to stay ahead of emerging coding challenges and ensure their models are tested against real-world, diverse development scenarios.
Disclaimer: The above content is generated by AI and is for reference only.