Mixture of Complementary Agents for Robust LLM Ensemble

Background

The study addresses a foundational step in multi-AI collaboration paradigms like model ensembling or debating: synthesizing outputs from multiple proposer large language models (LLMs) using a summarizer LLM. Existing selection strategies are myopic, typically prioritizing either the highest-accuracy individual models or enforcing variety, but they fail to account for the critical interactions between proposers themselves and between the proposers and the final summarizer. This oversight can lead to suboptimal final answers.

Key Points

The authors' central contribution is a fundamental reframing of the problem. They conceptualize proposer selection as a combinatorial selection problem analogous to feature selection in machine learning.

• Core Principle: The value of a proposer LLM is not defined in isolation but by its complementarity with other selected proposers and the summarizer. The goal is to find a subset of proposers whose combined information, when synthesized, yields the highest-quality output.
• Computational Challenge: Directly applying standard feature-selection algorithms is computationally prohibitive in the LLM context. Evaluating every possible combination of proposers requires extensive and costly LLM API calls or inference, making exhaustive search infeasible.
• Proposed Solution: The study bypasses the infeasible exhaustive search by investigating computationally feasible, greedy-style selection algorithms. These methods assess complementarity iteratively and efficiently using a small, labeled evaluation set, making them practical for real-world pipelines.
• Experimental Validation: The research validates that complementarity is an effective guiding principle for selection. Among the methods explored, certain greedy algorithms identify combinations that achieve superior performance relative to the computational cost of running the selected proposers.

Significance

This work shifts the focus from evaluating models in isolation to evaluating their synergy within a system. The significance is twofold:

1. Theoretical: It provides a principled, systems-oriented framework for multi-AI design, arguing that interactions, not just individual strengths, are paramount.
2. Practical: By developing and benchmarking efficient greedy selection methods, it offers actionable algorithms for practitioners to build more effective and cost-efficient multi-LLM pipelines, directly improving the final synthesized answers without requiring prohibitive computational resources.