Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
Introduces Strategic 16K, a leakage-controlled corpus of 16,000 diplomatic cables from WikiLeaks PlusD for document sensitivity classification Identifies and eliminates three categories of residual classification markers (label leakage) embedded within document bodies that inflate performance estimates BERT achieves the strongest performance on the clean benchmark (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%) TF-IDF with Logistic Regression emerges as the
Analysis
TL;DR
- Introduces Strategic 16K, a leakage-controlled corpus of 16,000 diplomatic cables from WikiLeaks PlusD for document sensitivity classification
- Identifies and eliminates three categories of residual classification markers (label leakage) embedded within document bodies that inflate performance estimates
- BERT achieves the strongest performance on the clean benchmark (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%)
- TF-IDF with Logistic Regression emerges as the best classical model, delivering strong results at significantly lower computational cost
- Establishes the first fully reproducible sensitivity classification benchmark constructed under explicit leakage-controlled conditions
Why It Matters
This research directly addresses a critical reliability gap in AI-driven document classification systems used in organizational and government contexts, where misclassification can lead to regulatory violations or security breaches. By exposing and mitigating label leakage—a pervasive but underreported issue—the study provides a more honest assessment of model capabilities, preventing overconfidence in systems that may fail in production. For practitioners building classification pipelines, it serves as a cautionary benchmark emphasizing data integrity over raw model architecture choices.
Technical Details
- Dataset: Strategic 16K comprises 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD), constructed with an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies
- Model architectures evaluated: Six models spanning classical machine learning and transformer-based approaches, including BERT, ELECTRA, TF-IDF with Logistic Regression, and others
- Performance on clean benchmark: BERT leads with 89.14% accuracy and 89.33% F1; ELECTRA follows at 88.57% accuracy and 88.90% F1; TF-IDF with Logistic Regression achieves the strongest classical results at significantly lower computational cost
- Methodology: Systematic benchmark comparing classical ML and transformer architectures under explicit leakage-controlled conditions, ensuring performance estimates reflect genuine content-based sensitivity signal learning rather than surface shortcuts
- arXiv reference: 2608.16928 [cs.LG], submitted 5 Aug 2026
Industry Insight
- Organizations deploying AI for document classification should prioritize data auditing and leakage detection before model selection; even state-of-the-art transformers can produce inflated metrics on contaminated datasets, leading to false confidence in production deployments
- Classical models like TF-IDF with Logistic Regression remain highly competitive for sensitivity classification tasks, offering a cost-effective alternative that may be preferable in resource-constrained environments or when interpretability is valued
- The Strategic 16K benchmark and its leakage removal protocol should be adopted as a standard evaluation practice in the field, encouraging the community to report performance on cleaned data rather than raw, potentially contaminated corpora
Disclaimer: The above content is generated by AI and is for reference only.