FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification
FrenchNews-7 introduces a cross-publisher benchmark for classifying French-language news articles into seven editorial desk categories using a multi-outlet corpus and URL-derived taxonomy A hybrid labeling pipeline combines publisher URL slugs with LLM annotation for ambiguous cases, achieving strong inter-rater reliability (human-human κ = 0.806, pairwise κ ≥ 0.766) CamemBERT-base fine-tuned on full article text emerges as the strongest model, achieving 0.799 overall recall and outperforming ze
Analysis
TL;DR
- FrenchNews-7 introduces a cross-publisher benchmark for classifying French-language news articles into seven editorial desk categories using a multi-outlet corpus and URL-derived taxonomy
- A hybrid labeling pipeline combines publisher URL slugs with LLM annotation for ambiguous cases, achieving strong inter-rater reliability (human-human κ = 0.806, pairwise κ ≥ 0.766)
- CamemBERT-base fine-tuned on full article text emerges as the strongest model, achieving 0.799 overall recall and outperforming zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B)
- Cross-publisher generalization is uneven: Sport, Culture & Loisirs, and International transfer cleanly, while Economie (recall = 0.517) and Societe (precision = 0.577) expose inherent editorial boundary ambiguity
- The work demonstrates that some classification failures reflect genuine editorial convention disagreements rather than model limitations, establishing a reliability-tier guidance framework
Why It Matters
This benchmark addresses a critical gap in multilingual NLP by providing a rigorously evaluated resource for French-language news classification, a domain where cross-publisher generalization remains underexplored. The finding that certain categories hit a performance ceiling due to ambiguous editorial boundaries—not model capacity—offers practitioners a realistic framework for setting expectations when deploying text classification systems in real-world newsroom environments.
Technical Details
- Dataset: FrenchNews-7 is a large multi-outlet French-language news corpus with a seven-class taxonomy derived from publisher URL structures, covering categories such as Sport, Culture & Loisirs, International, Economie, and Societe
- Labeling pipeline: Hybrid approach combining deterministic publisher URL slug extraction with LLM-based annotation for structurally ambiguous cases, validated through an inter-rater study involving 2 humans and 2 LLMs
- Model evaluation: Compares lexical, multilingual, and French-specific classifiers (including CamemBERT variants) under both in-distribution and held-out-publisher settings, with zero-shot LLM baselines tested on the held-out pool
- Key results: CamemBERT-base on full article text achieves 0.799 overall recall, surpassing headline-only input and all zero-shot LLM baselines; Economie and Societe categories show the largest performance gaps, with Economie recall (0.517) closely matching blinded human agreement (0.55)
- Reproducibility: Fine-tuned model, labeled manifest, collection scripts, and reliability-tier guidance table are publicly released
Industry Insight
- News organizations and media analytics firms should treat editorial desk classification as a bounded problem: categories like Economie and Societe may inherently resist high-accuracy automation due to genuine structural ambiguity in French journalism conventions
- The cross-publisher evaluation methodology provides a template for assessing model generalization in other language-specific and domain-specific classification tasks where training and deployment outlets differ
- Practitioners should adopt reliability-tier frameworks when deploying classifiers, explicitly communicating to stakeholders which categories are production-ready versus which require human oversight due to irreducible boundary ambiguity
Disclaimer: The above content is generated by AI and is for reference only.