GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
GUI-CC is a new benchmark that evaluates contextual consistency of GUI world models as multi-step agent environments, addressing the mismatch between one-step next-screen prediction and real-world agent use The benchmark features two complementary tracks: an offline reference-action track (500 tasks from GUIOdyssey) and an online agent-loop track (200 emulator-verified tasks across 30 mobile apps) GUI-CC evaluates four key metrics: transition fidelity, transition plausibility, contextual consist
Analysis
TL;DR
- GUI-CC is a new benchmark that evaluates contextual consistency of GUI world models as multi-step agent environments, addressing the mismatch between one-step next-screen prediction and real-world agent use
- The benchmark features two complementary tracks: an offline reference-action track (500 tasks from GUIOdyssey) and an online agent-loop track (200 emulator-verified tasks across 30 mobile apps)
- GUI-CC evaluates four key metrics: transition fidelity, transition plausibility, contextual consistency, and task progress
- Experiments reveal that plausible single-step generation does not guarantee reliable multi-step environment simulation; current models fail to preserve task-relevant context or support executable rollouts
Why It Matters
This benchmark exposes a critical gap in how GUI world models are evaluated versus how they are actually used — as multi-step environments for autonomous agents. For AI practitioners building GUI agents, it highlights that next-screen prediction quality alone is insufficient; contextual consistency across repeated interactions is equally vital for reliable agent performance.
Technical Details
- Two-track evaluation framework: The offline track rolls models along real mobile GUI trajectories from GUIOdyssey (500 tasks), while the online track deploys fixed probing agents to interact with model-generated UIs across 30 mobile apps (200 emulator-verified tasks)
- Four evaluation metrics: Transition fidelity (accuracy of state transitions), transition plausibility (visual realism of generated screens), contextual consistency (preservation of task-relevant state across steps), and task progress (ability to advance toward goals over multi-step rollouts)
- Core finding: Current GUI world models produce visually plausible single-step outputs but degrade significantly when reused iteratively, failing to maintain coherent context or support executable multi-step agent trajectories
Industry Insight
- Benchmark designers and model developers should shift evaluation focus from isolated next-screen prediction to multi-step consistency, as this better reflects real agent deployment requirements
- The gap between single-step plausibility and multi-step reliability suggests that training objectives for GUI world models need to incorporate long-horizon consistency losses, not just per-step fidelity
- For agent system builders, relying on current GUI world models for autonomous multi-step tasks carries significant risk of context drift; validation on benchmarks like GUI-CC should be a prerequisite before production deployment
Disclaimer: The above content is generated by AI and is for reference only.