Reliability-Aware Sexism Detection: Combining DPO with Annotator Agreement and Token-Level Confidence Scoring
RA-DPO integrates annotator agreement, model confidence, and token-level uncertainty into a single reliability score for preference optimization Training on only the top 30% most reliable preference pairs matches full-data DPO performance, demonstrating significant cost reduction Selective prediction achieves 96.2% accuracy at 50% coverage (true-agreement) and 88.7% (predicted-agreement), both surpassing the 85.3% no-agreement baseline The approach enables inference-time abstention, trading cove
Analysis
TL;DR
- RA-DPO integrates annotator agreement, model confidence, and token-level uncertainty into a single reliability score for preference optimization
- Training on only the top 30% most reliable preference pairs matches full-data DPO performance, demonstrating significant cost reduction
- Selective prediction achieves 96.2% accuracy at 50% coverage (true-agreement) and 88.7% (predicted-agreement), both surpassing the 85.3% no-agreement baseline
- The approach enables inference-time abstention, trading coverage for accuracy in subjective classification tasks
- Evaluated on 6,920 multilingual posts from EXIST 2023 with fine-tuning of gpt-4o and validation on Llama and Qwen 3B models
Why It Matters
This work addresses a fundamental limitation in subjective NLP classification: the loss of annotation uncertainty when collapsing multi-annotator labels into majority decisions. For AI practitioners building safety or content moderation systems, RA-DPO offers a practical framework to improve reliability without additional labeling costs, while providing a deployable abstention mechanism that can reduce harmful false positives in high-stakes applications.
Technical Details
- RA-DPO (Reliability-Aware Direct Preference Optimization) combines three signals—annotator agreement, model confidence, and token-level uncertainty—into a unified reliability score used to rank preference pairs during DPO training
- Data: 6,920 multilingual posts from the EXIST 2023 shared task on sexism detection, with multiple annotator labels per instance
- Training: Fine-tuned OpenAI gpt-4o base model via DPO; validated generalization on two open-weight 3B models (Llama and Qwen)
- Selective prediction: At inference, the model can abstain on low-reliability samples, achieving 96.2% accuracy at 50% coverage under true-agreement conditions and 88.7% under the more realistic predicted-agreement setting, compared to 85.3% for a baseline ignoring agreement
- Key finding: Training exclusively on the top 30% most reliable preference pairs achieves performance parity with full-data DPO, indicating that reliability-aware filtering can substantially reduce training compute and cost
Industry Insight
- Content moderation and safety classification systems should move beyond majority-vote labels and incorporate annotator disagreement as a first-class signal, particularly for subjective categories like sexism, hate speech, and harassment
- Selective prediction with abstention is a deployable strategy for reducing false positives in production moderation pipelines, where the cost of erroneous decisions often outweighs the cost of missed detections
- Reliability-aware data selection during preference optimization could become a standard technique for efficient fine-tuning across subjective NLP tasks, offering both cost savings and improved deployment safety
Disclaimer: The above content is generated by AI and is for reference only.