Show HN: WorldBuild Bench repo: testing LLM world coherence with 3D games
WorldBuild Bench introduces a standardized framework for evaluating AI agents' ability to maintain spatial, temporal, and causal coherence in 3D environments, moving beyond simple code execution checks. The benchmark utilizes a fixed technical stack (Three.js, Rapier physics, Playwright) and identical design briefs to isolate model performance, generating 27 playable 3D games from nine different AI models. Automated metrics like Playability and World Coherence Scores proved insufficient for capt
Analysis
TL;DR
- WorldBuild Bench introduces a standardized framework for evaluating AI agents' ability to maintain spatial, temporal, and causal coherence in 3D environments, moving beyond simple code execution checks.
- The benchmark utilizes a fixed technical stack (Three.js, Rapier physics, Playwright) and identical design briefs to isolate model performance, generating 27 playable 3D games from nine different AI models.
- Automated metrics like Playability and World Coherence Scores proved insufficient for capturing quality, as they failed to detect fundamental gameplay flaws that human players immediately noticed.
- The primary evaluation method is a blind human preference arena using Bradley-Terry Elo ratings, acknowledging that "game feel" and coherence are best measured by human interaction rather than static code analysis.
- Significant variance in cost and efficiency was observed, with some models costing over $250 per game while others cost around $6, highlighting economic disparities in current AI coding capabilities.
Why It Matters
This initiative addresses a critical gap in AI evaluation by shifting focus from syntactic correctness to semantic and experiential coherence in complex, interactive systems. For AI practitioners, it demonstrates that traditional automated benchmarks are inadequate for assessing generative agents in dynamic worlds, necessitating human-in-the-loop evaluation methods. The project also provides a reproducible, open-source harness that allows researchers to compare model performance on realistic, multi-constraint tasks involving physics, AI, and user interface design.
Technical Details
- Standardized Harness: Uses a provider-agnostic TypeScript agent loop (~300 lines) with raw fetch adapters, ensuring no vendor-specific SDKs influence results. The environment includes Three.js for rendering, Rapier for physics, and Playwright for automated testing.
- Evaluation Metrics: Implements two machine-computed scores: Playability Score (checks for fatal errors, rendering, input responsiveness, and stable FPS) and World Coherence Score (probes for finite positions, sane cameras, and state evolution via scripted tests).
- Human-Centric Ranking: Employs a blind A/B testing arena where users compare games built from the same brief. Ratings are aggregated using a Bradley-Terry model to produce an Elo-like scale, with provisional status until sufficient comparison data is collected.
- Task Design: Features three distinct genres—Arena Combat, Physics Puzzle, and Racing—each with a concise Game Design Document (GDD) to constrain creative divergence and focus on execution quality.
- Cost and Efficiency Tracking: Publishes generation time, token cost, and code size for transparency, noting that costs ranged widely from ~$6 to ~$252 per game depending on the model used.
Industry Insight
- Shift to Experiential Evaluation: The industry must move beyond static code benchmarks for generative AI agents. Future evaluation frameworks need to incorporate dynamic, interactive testing environments where human feedback is central to assessing quality, especially for applications requiring spatial and causal reasoning.
- Cost-Benefit Analysis in AI Development: The wide disparity in costs ($6 vs. $252) suggests that efficiency optimizations are just as important as capability improvements. Developers should prioritize models that offer a balance of coherence and cost-effectiveness for production-grade applications.
- Importance of Standardized Baselines: By fixing the tools and briefs, WorldBuild Bench proves that controlled variables are essential for fair comparison. Adopting similar standardized harnesses in other domains could help reduce noise in AI performance reporting and accelerate progress tracking.
Disclaimer: The above content is generated by AI and is for reference only.