Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita
Introduction of Incognita, a framework separating social interaction from grounded execution to evaluate agents in socially distributed task environments. Definition of socially distributed tasks where knowledge is partitioned across role-isolated participants, requiring communication for exploration and actions for exploitation. Evaluation of three generative models on 18 retail tasks reveals low reliability despite progress, with success rates rising from 0% to 17.2%. Findings indicate that st
Analysis
TL;DR
- Introduction of Incognita, a framework separating social interaction from grounded execution to evaluate agents in socially distributed task environments.
- Definition of socially distributed tasks where knowledge is partitioned across role-isolated participants, requiring communication for exploration and actions for exploitation.
- Evaluation of three generative models on 18 retail tasks reveals low reliability despite progress, with success rates rising from 0% to 17.2%.
- Findings indicate that stronger models elicit more hidden knowledge and contact more entities, but premature finalization remains a significant issue.
- The study highlights that such environments expose critical behavioral gaps like source selection and grounded action attempts before achieving reliable success.
Why It Matters
This research addresses a critical gap in AI evaluation by combining the persistent state of grounded benchmarks with the rich interactions of social simulations. For practitioners, it provides a rigorous method to test agent reliability in complex scenarios where information is siloed and actions have real consequences. The findings serve as a baseline for improving agent coordination and decision-making in multi-agent systems.
Technical Details
- Framework Architecture: Incognita is built on Concordia and decouples social interaction from grounded execution. It uses a routing mechanism where agents send messages to users or specialist entities, which then mediate operations in a deterministic sub-environment.
- Evaluation Methodology: The study utilizes Incognita-Retail, adapting tau-bench retail into a multi-entity environment. It preserves final-state reward semantics while introducing role-isolated participants.
- Experimental Setup: Three generative agent models were tested across 18 tasks stratified by social breadth, totaling 540 trials. Metrics included success rate, premature finalization, and behavioral indicators like knowledge elicitation.
- Performance Metrics: Success rates improved from 0% to 8.9% and 17.2% across models. Premature finalization decreased from 100% to 87% and 58%, indicating better adherence to task protocols but still significant errors.
- Behavioral Analysis: Stronger models demonstrated increased attempts at grounded writes and contact with more entities, suggesting better exploration capabilities, though overall reliability remained low.
Industry Insight
- Benchmark Design: Developers should incorporate socially distributed elements into future benchmarks to better simulate real-world complexity where information is not centrally available.
- Agent Reliability: The high rate of premature finalization suggests current models struggle with patience and verification; improving this requires better incentive structures or architectural changes to delay action until sufficient knowledge is gathered.
- Multi-Agent Coordination: As agents become more capable, the focus must shift from individual performance to inter-agent communication efficiency and trust mechanisms in partitioned knowledge environments.
Disclaimer: The above content is generated by AI and is for reference only.