Evaluating Large Language Models in a Complex Hidden Role Game

Background

The research aims to assess the deceptive potential of LLMs through a controlled social deduction game setting—Secret Hitler—to understand their capabilities and limitations in complex reasoning tasks. This involves developing novel metrics and benchmarking against both rule-based algorithms and human performance.

Key Points

• Metrics Introduced:

  • Role Identification Accuracy: Measures how well the model can correctly identify player roles.
  • Deception Retention Rate: Evaluates the ability to maintain deceptive strategies over multiple rounds.
  • Game State Impact Rate: Assesses the influence of the model’s actions on the game's outcome.

• Performance Comparison:

  • Models like Llama 3.1 70B showed significantly lower accuracy (59.7%) compared to rule-based agents, which aligned with expert human decisions 86.7% of the time.
  • Models playing as Fascists performed poorly in sustaining deception, leading to shorter game durations and negative impact scores.

• Reasoning Techniques:

  • Chain-of-Thought prompting and internal memory enhancements did not improve performance; instead, they led to a 23.2% decrease in win rates for fascist roles.

Significance

The study highlights the gap between conversational fluency and strategic reasoning capabilities of current LLMs. It underscores that while these models can engage in detailed dialogue, their ability to sustain complex deception over multiple turns remains limited. This finding is crucial for AI safety research, as it indicates that despite advancements, existing architectures struggle with sophisticated multi-turn manipulation tasks.

Key Insights:

• Complex Strategic Tasks: Current LLMs are not yet capable of mastering complex strategic reasoning required in games like Secret Hitler.
• Future Implications: As LLM capabilities increase, identifying when they begin to effectively perform such tasks will be critical for ensuring AI safety and alignment. The developed framework provides a robust benchmarking tool for future research.

The study's open-source nature makes it a valuable resource for researchers looking to further explore the deceptive potential of LLMs and develop more effective alignment strategies.