Stop Ranking Agent Configs by Average Score
Standard average scoring masks critical interaction effects between model, prompt, and tool configurations in agent evaluation. Best-Worst Scaling (MaxDiff) combined with Plackett-Luce ranking models provides superior signal for comparing agent configurations. The study demonstrates that prompt-tool interactions significantly impact performance, independent of raw model capability. Automated evaluation agents can execute this comparative framework to generate actionable deployment recommendation
Analysis
TL;DR
- Standard average scoring masks critical interaction effects between model, prompt, and tool configurations in agent evaluation.
- Best-Worst Scaling (MaxDiff) combined with Plackett-Luce ranking models provides superior signal for comparing agent configurations.
- The study demonstrates that prompt-tool interactions significantly impact performance, independent of raw model capability.
- Automated evaluation agents can execute this comparative framework to generate actionable deployment recommendations.
- Shipping the "best average" configuration often leads to suboptimal production outcomes compared to interaction-aware selection.
Why It Matters
This approach resolves the common pitfall where teams select agent configurations based on isolated metric averages, ignoring how components interact. By adopting comparative ranking methods like Best-Worst Scaling, practitioners can identify robust configurations that perform well under real-world variability rather than just theoretical peaks. This leads to more reliable agent deployments and more efficient iteration cycles in production environments.
Technical Details
- Evaluation Method: Utilizes Best-Worst Scaling (MaxDiff), where a judge model selects only the single best and single worst output from a batch of anonymized configurations, avoiding arbitrary 1-10 scoring scales.
- Statistical Modeling: Fits a Plackett-Luce model or similar utility ranking algorithm to convert pairwise comparisons into utility scores for each configuration and interaction term.
- Experimental Design: Tested on the
shubh303/Invoice-to-Jsondataset (100 invoices) with a factorial grid of 8 configurations varying three binary factors: Model (Claude Haiku 4.5 vs. GPT-5.4-mini), Prompt (Systematic Planner vs. Contextual Leaper), and Tool (stream table rows vs. semantic search). - Judge Model: Used Claude Sonnet 4.6 as the evaluator to determine best/worst outputs for each batch of five randomly sampled configurations per invoice.
- Interaction Analysis: The method highlights specific interactions, such as the "Contextual Leaper" prompt pairing effectively with semantic search tools, which average scores failed to capture accurately.
Industry Insight
- Move away from leaderboard-style metrics that prioritize absolute scores over relative performance; focus on comparative advantage within specific operational contexts.
- Implement automated evaluation loops where agent systems propose, test, and rank their own configurations using comparative methods to reduce human-in-the-loop overhead.
- Prioritize understanding component interactions (e.g., how a specific prompt affects a specific tool) during the design phase, as these synergies often outweigh the marginal gains of upgrading base models.
Disclaimer: The above content is generated by AI and is for reference only.