Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
Chain-of-Models (CoM) introduces an automated audit pipeline where a second model inspects a first model's reasoning trace before delivering a final judgment, addressing the vulnerability of LLM judges to cognitive biases Standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases but performs poorly as an auditor for Qwen2.5-72B's biased traces The optimal auditor is bias-specific: GPT-4o excels on bandwagon, authority, and dist
Analysis
TL;DR
- Chain-of-Models (CoM) introduces an automated audit pipeline where a second model inspects a first model's reasoning trace before delivering a final judgment, addressing the vulnerability of LLM judges to cognitive biases
- Standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases but performs poorly as an auditor for Qwen2.5-72B's biased traces
- The optimal auditor is bias-specific: GPT-4o excels on bandwagon, authority, and distraction biases, while GLM-5 is strongest on sycophancy
- A per-bias auditor selection rule scoring candidates on functional diversity, per-bias standalone resistance, and calibrated audit effectiveness achieves 0.884 accuracy across four biased slices, outperforming the best fixed auditor (0.824) and the no-audit baseline (0.805)
- The authors release data, configurations, and an LLM-agent skill to enable reproducibility and further research
Why It Matters
This work directly addresses a critical reliability gap in AI evaluation: as LLMs increasingly serve as automated judges, their susceptibility to cognitive biases undermines the trustworthiness of AI-generated assessments. The findings challenge the intuitive assumption that the strongest model is the best auditor, offering practitioners a principled, bias-aware framework for selecting auditors that can significantly improve judgment accuracy without relying on brittle prompt engineering or costly human evaluation.
Technical Details
- Chain-of-Models (CoM) architecture: A two-stage pipeline where Model A produces a reasoning trace and judgment, and Model B (the auditor) inspects the trace before producing the final judgment; the study compares same-model, same-family, and cross-family auditor configurations
- Evaluation scope: 9 models from 6 families (including Kimi-K2.5, Qwen2.5-72B, GPT-4o, GLM-5), 4 cognitive biases (bandwagon, authority, distraction, sycophancy), and 4 factual datasets
- Auditor selection rule: A scoring function that ranks candidate auditors along three dimensions—functional diversity (architectural/behavioral differences from the judge), per-bias standalone resistance, and calibrated audit effectiveness measured on a held-out calibration set
- Quantitative results: The bias-aware selector achieves 0.884 accuracy across biased slices versus 0.824 for the strongest single fixed auditor and 0.805 for the no-audit baseline, demonstrating a 6-point gain over the best fixed auditor
- Open science contribution: Data, configurations, and an LLM-agent skill are released to support community adoption and further auditing research
Industry Insight
- Organizations deploying LLM judges for evaluation, content moderation, or RLHF should adopt bias-aware auditor selection rather than defaulting to their strongest model; matching auditor identity to the specific bias type can yield meaningful accuracy gains
- The decoupling of standalone performance from audit effectiveness suggests that model diversity—rather than raw capability—is a key design principle for robust multi-model auditing pipelines
- As LLM judges become more prevalent in production, bias-specific auditing should be treated as a standard quality assurance layer, with per-bias auditor selection rules offering a scalable alternative to human-in-the-loop evaluation
Disclaimer: The above content is generated by AI and is for reference only.