Leak It: A Probabilistic Approach to Training-Data Extraction from Black-Box Language Models
Aggregate ROC-AUC metrics for membership inference attacks on language models are misleading because blind bag-of-words baselines already achieve high AUC (0.97 on WikiMIA) without any model sampling, making sampling-based attacks appear effective when they are not. Black-box sampling can verbatim extract training data for a tail of documents that blind attacks cannot reach; on Pythia-6.9B, 16.6% of Pile documents with real identifiers were exactly reproduced and confirmed attributable to those
Analysis
TL;DR
- Aggregate ROC-AUC metrics for membership inference attacks on language models are misleading because blind bag-of-words baselines already achieve high AUC (0.97 on WikiMIA) without any model sampling, making sampling-based attacks appear effective when they are not.
- Black-box sampling can verbatim extract training data for a tail of documents that blind attacks cannot reach; on Pythia-6.9B, 16.6% of Pile documents with real identifiers were exactly reproduced and confirmed attributable to those documents via mismatched-prefix controls.
- Per-document disclosure scales with model capacity (5.6% to 16.6% from Pythia-410M to Pythia-6.9B) and is domain-dependent: identifier leakage is ~3x stronger in code than prose, and held-out continuation recovery is confined to code (+0.44 member gap on GitHub vs +0.014 on prose).
- Sampling parameters (temperature, nucleus) have minimal impact on leakage, 16-token prefixes suffice for extraction, and corpus deduplication provides no measurable reduction in risk.
- The authors release "leakit," a black-box extraction-audit tool, and argue privacy audits should report per-document extraction rates decomposed by domain rather than a single aggregate AUC.
Why It Matters
This work fundamentally challenges how the AI security community evaluates membership inference attacks, showing that widely cited aggregate metrics obscure real, document-level privacy harms that persist even when statistical tests appear non-significant. For AI practitioners deploying language models, it demonstrates that training-data leakage is not merely a statistical artifact but a concrete risk that scales with model size and varies significantly across domains, with code being particularly vulnerable.
Technical Details
- The paper reframes black-box sampling-based leakage as a probabilistic problem, treating N samples from p(.|x) as estimates of the output distribution and casting leakage signals as functionals of that distribution.
- On WikiMIA, a blind bag-of-words classifier achieves AUC 0.97 (TPR 0.90 at 5% FPR), and adding sampling provides no statistical improvement; on the IID Pile split (MIMIR), neither self-concentration nor gold-continuation recovery significantly outperforms the blind baseline (incremental AUC 95% CI includes zero).
- Per-document analysis on Pythia-6.9B reveals 83 of 500 Pile documents with real identifiers (16.6%; 21.3% of those with email addresses) had exact identifiers reproduced and confirmed via mismatched-prefix controls, isolating leaks to specific documents rather than globally common strings.
- Domain decomposition shows identifier leakage grows with capacity in both code (5.6% to 16.6%) and prose (4.0% to 12.1%), while arbitrary continuation recovery is nearly absent in prose (+0.014 max) but substantial in code (+0.44 member gap on GitHub).
- Robustness checks find temperature and nucleus sampling parameters have little effect, 16-token prefixes are sufficient for extraction, and deduplication of the training corpus does not reduce leakage.
Industry Insight
- Privacy audits for language models should abandon single aggregate AUC reports in favor of per-document extraction rates broken down by domain, as the latter reveals harms that aggregate metrics completely hide.
- Organizations training on code-heavy datasets should treat model capacity scaling as a direct privacy risk multiplier, since code exhibits significantly stronger leakage signals than prose across all measured dimensions.
- Deduplication alone is insufficient as a privacy mitigation; the absence of any measurable reduction from corpus deduplication suggests that complementary techniques (e.g., differential privacy, targeted sanitization of high-risk document tails) are necessary to address black-box extraction risks.
Disclaimer: The above content is generated by AI and is for reference only.