Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
Proposes an information-weighted cross-entropy loss that rescales token-level contributions using TF-IDF statistics to reduce memorization in LLMs Experiments across five decoder-only models (1.1B–13B parameters) show consistent reductions in memorized substring length while preserving perplexity and downstream task performance TF-IDF weighting reduces average substring memorization length by 14% under LoRA fine-tuning and by 58% under full-weight fine-tuning on TinyLLaMA 1.1B The approach is ar
Analysis
TL;DR
- Proposes an information-weighted cross-entropy loss that rescales token-level contributions using TF-IDF statistics to reduce memorization in LLMs
- Experiments across five decoder-only models (1.1B–13B parameters) show consistent reductions in memorized substring length while preserving perplexity and downstream task performance
- TF-IDF weighting reduces average substring memorization length by 14% under LoRA fine-tuning and by 58% under full-weight fine-tuning on TinyLLaMA 1.1B
- The approach is architecture-agnostic and adds less than 3% computational overhead to existing training pipelines
Why It Matters
This work addresses a critical and growing concern in the AI industry: LLM memorization of training data, which raises copyright, privacy, and safety risks. By introducing a simple, lightweight modification to the training objective, it offers practitioners an easy-to-deploy solution that does not require architectural changes or significant computational resources, making it accessible for both research and production environments.
Technical Details
- TF-IDF Weighted Loss: Replaces uniform token weighting in cross-entropy loss with TF-IDF-derived weights, down-weighting frequent, low-information tokens and up-weighting semantically informative ones
- Model Scope: Evaluated on five decoder-only LLMs ranging from 1.1B to 13B parameters, demonstrating scalability across model sizes
- Fine-tuning Regimes: Tested under both LoRA (parameter-efficient) and full-weight fine-tuning, with memorization reductions of 14% and 58% respectively on TinyLLaMA 1.1B
- Performance Preservation: Maintains perplexity and downstream task performance while reducing memorization, indicating no trade-off between safety and capability
- Computational Efficiency: Adds less than 3% overhead, as TF-IDF statistics are precomputed and applied as simple multiplicative weights during loss calculation
Industry Insight
- The memorization problem in LLMs is likely to face increasing regulatory scrutiny; lightweight loss-level interventions like TF-IDF weighting could become standard practice in responsible model training pipelines
- The architecture-agnostic nature of this approach means it can be adopted across diverse model families (GPT, LLaMA, Mistral, etc.) without vendor lock-in or re-engineering
- The 58% memorization reduction under full fine-tuning suggests that for safety-critical applications, full fine-tuning with TF-IDF weighting may be preferable to parameter-efficient methods, though the trade-off with computational cost should be evaluated per use case
Disclaimer: The above content is generated by AI and is for reference only.