Stability-Aware Feature Design for Robust Watermark Detection in Machine-Generated Text
Introduces Pattern Stability Score (PSS), a novel detection framework that combines global/local z-score features with higher-order run-length statistics and autocorrelation signals to detect watermarks in LLM-generated text PSS achieves over 10-15 percentage point AUC improvements compared to prior z-score thresholding baselines and state-of-the-art deep learning methods across varying token lengths A single universal classifier generalizes across different LLMs (Llama-3-8B, Qwen2-7B), paraphra
Analysis
TL;DR
- Introduces Pattern Stability Score (PSS), a novel detection framework that combines global/local z-score features with higher-order run-length statistics and autocorrelation signals to detect watermarks in LLM-generated text
- PSS achieves over 10-15 percentage point AUC improvements compared to prior z-score thresholding baselines and state-of-the-art deep learning methods across varying token lengths
- A single universal classifier generalizes across different LLMs (Llama-3-8B, Qwen2-7B), paraphrasers (Mistral-7B, Qwen2-7B, Gemma-7B), and text domains without retraining, maintaining above 87.8% AUC
- The framework is stress-tested under up to eight rounds of paraphrasing, addressing the critical vulnerability of existing detectors to paraphrasing attacks and short texts
- Evaluated on three benchmark datasets (PG-19, CNN/DailyMail, WikiText) demonstrating robustness across diverse domains and text lengths
Why It Matters
Watermark detection for machine-generated text is increasingly critical as LLM adoption accelerates, yet current methods degrade sharply under paraphrasing and on short texts—two common evasion tactics. This work provides a more robust, generalizable detection framework that could become a practical standard for content provenance verification in academic, media, and enterprise settings.
Technical Details
- Feature Engineering: PSS combines global and local z-score features with higher-order statistics of run-length patterns, enriched by autocorrelation signals and stability scores computed across paraphrase depth, capturing both static and dynamic watermark signatures
- Stability Dynamics: The framework computes stability scores over multiple paraphrase iterations, leveraging the observation that genuine watermark patterns persist across paraphrasing while noise-based artifacts degrade
- Universal Classification: A single classifier trained on one set of LLMs, paraphrasers, and domains generalizes without retraining to entirely different configurations, achieving above 87.8% AUC in cross-domain experiments
- Evaluation: Benchmarked on PG-19, CNN/DailyMail, and WikiText using Llama-3-8B and Qwen2-7B as generators, with Mistral-7B, Qwen2-7B, and Gemma-7B as paraphrasers, stress-tested up to eight paraphrasing rounds
- Performance Gains: Improves detection AUC by over 10-15 percentage points compared to prior z-score thresholding baselines and leading deep learning methods, with consistent gains across different token lengths
Industry Insight
- Organizations deploying LLM-generated content should consider adopting stability-aware detection frameworks rather than single-pass z-score methods, as paraphrasing-based evasion is increasingly accessible and effective against weaker detectors
- The universal generalization property of PSS suggests that content verification systems may not need per-model or per-domain retraining, significantly reducing operational costs for platform-scale deployment
- As watermark robustness becomes a competitive differentiator, expect increased investment in multi-round paraphrasing stress tests as standard evaluation protocols for any new detection method
Disclaimer: The above content is generated by AI and is for reference only.