Inter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended Text
Proposes CorrGap, a novel metric to quantify inter-dimension dependence in LLM judges by measuring correlation differences between predicted and ground truth scores across text groups Demonstrates that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks Introduces DimCheck, an iterative method that removes unrelated evidence from chain-of-thought reasoning step-wise to mitigate cross-dimensional contamination DimCheck outperforms strong baselines across
Analysis
TL;DR
- Proposes CorrGap, a novel metric to quantify inter-dimension dependence in LLM judges by measuring correlation differences between predicted and ground truth scores across text groups
- Demonstrates that inter-dimension dependence is pervasive across LLM judges in open-ended text evaluation tasks
- Introduces DimCheck, an iterative method that removes unrelated evidence from chain-of-thought reasoning step-wise to mitigate cross-dimensional contamination
- DimCheck outperforms strong baselines across three LLMs and four evaluation tasks
- Smaller trained LLMs can approximate larger models in DimCheck, significantly reducing inference costs
Why It Matters
This research directly addresses a critical reliability concern in LLM-as-a-judge frameworks, which are increasingly the standard for evaluating open-ended text generation. By exposing and mitigating inter-dimension dependence, practitioners can achieve more trustworthy and dimension-specific evaluations, which is essential for fair model comparison and alignment research.
Technical Details
- CorrGap: A metric that quantifies inter-dimension dependence by computing the difference in correlations between LLM-predicted scores and ground truth scores across different groups of texts, isolating how much non-target dimensions influence target dimension evaluations.
- DimCheck: An iterative chain-of-thought refinement method that step-wise removes evidence unrelated to the target evaluation dimension, effectively decoupling cross-dimensional reasoning contamination in LLM judges.
- Evaluation scope: Validated across three LLMs and four distinct open-ended text evaluation tasks, demonstrating consistent improvement over strong baselines.
- Efficiency finding: Smaller trained LLMs can approximate the performance of larger models when used within the DimCheck framework, offering a practical path toward cost-effective evaluation pipelines.
Industry Insight
- Organizations relying on LLM-as-a-judge for model evaluation should audit their judges for inter-dimension dependence, as undetected cross-contamination can systematically bias multi-dimensional assessments.
- DimCheck's step-wise evidence removal approach offers a reusable template for improving reasoning fidelity in any multi-criteria LLM evaluation pipeline.
- The finding that smaller models can approximate larger ones in DimCheck suggests that high-quality evaluation need not require proportionally large models, opening the door to more accessible and cost-efficient evaluation infrastructure.
Disclaimer: The above content is generated by AI and is for reference only.