Investigating Knowledge Transfer Across Interactive Dialogue Games
The paper investigates how knowledge transfers across different dialogue games by finetuning LLMs on games from the clembench suite A task-transferability graph derived via binary integer optimization reveals that some games benefit more from transfer learning than direct finetuning The visuospatial family of games (e.g., exploration games) demonstrates the strongest transferability across tasks Task vector analysis captures game-role relationships but fails to reveal meaningful transferability
Analysis
TL;DR
- The paper investigates how knowledge transfers across different dialogue games by finetuning LLMs on games from the clembench suite
- A task-transferability graph derived via binary integer optimization reveals that some games benefit more from transfer learning than direct finetuning
- The visuospatial family of games (e.g., exploration games) demonstrates the strongest transferability across tasks
- Task vector analysis captures game-role relationships but fails to reveal meaningful transferability patterns, indicating the need for more sophisticated evaluation metrics
Why It Matters
This research addresses a critical question for AI practitioners: whether training language models on one interactive dialogue task can meaningfully improve performance on related but distinct tasks. As dialogue-based AI systems become more prevalent in gaming, education, and human-computer interaction, understanding cross-task knowledge transfer can significantly reduce training costs and accelerate model development. The findings also challenge the assumption that cosine similarity of task vectors is sufficient to predict transferability.
Technical Details
- The study uses the clembench suite (Chalamalasetti et al., 2023), a collection of interactive dialogue games requiring complex cognitive skills and coordination between players
- Two complementary analyses were performed: (i) a task-transferability graph constructed using a binary integer optimization program (Zamir et al., 2018) with task performance as the primary metric, and (ii) task vector computation (Ilharco et al., 2022) to study similarities across finetuned models
- LLM models were finetuned on individual dialogue games, then evaluated for transfer performance on other games in the suite
- The visuospatial game family showed the highest transferability, suggesting that spatial reasoning and exploration skills generalize well across dialogue game contexts
- Task vector similarity was found to capture structural game-role relationships but almost no actual transferability patterns, highlighting a limitation of current similarity-based metrics
Industry Insight
- Developers building multi-task dialogue agents should prioritize visuospatial and exploration-based training data, as these yield the broadest transfer benefits across downstream game tasks
- The failure of task vector similarity to predict transferability suggests the industry should invest in more nuanced metrics for evaluating cross-task generalization, rather than relying solely on vector-space proximity
- For cost-efficient model training, transfer-based fine-tuning strategies may outperform direct finetuning on target tasks for certain dialogue game categories, offering a practical pathway to reduce compute requirements
Disclaimer: The above content is generated by AI and is for reference only.