KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Training
KItCAT introduces a lightweight training strategy that injects specialized knowledge into decoder-only LLMs by stochastically corrupting input sequences during auto-regressive training Random subsets of input tokens are replaced with other vocabulary tokens while preserving original next-token labels, generating diverse training samples at negligible computational cost The approach eliminates the need for expensive paraphrase generation, which prior methods relied on to address the low-repetitio
Analysis
TL;DR
- KItCAT introduces a lightweight training strategy that injects specialized knowledge into decoder-only LLMs by stochastically corrupting input sequences during auto-regressive training
- Random subsets of input tokens are replaced with other vocabulary tokens while preserving original next-token labels, generating diverse training samples at negligible computational cost
- The approach eliminates the need for expensive paraphrase generation, which prior methods relied on to address the low-repetition problem in niche documents
- KItCAT consistently outperforms standard Continued Pre-Training (CPT) across multiple datasets and model families
- Code has been made publicly available, enabling reproducibility and further research in knowledge injection techniques
Why It Matters
This work addresses a critical bottleneck in domain adaptation of LLMs: efficiently injecting specialized knowledge from niche sources without relying on computationally expensive paraphrasing pipelines. For AI practitioners fine-tuning models on technical manuals, domain-specific documents, or proprietary data, KItCAT offers a cost-effective alternative that scales well. The approach is particularly relevant for organizations with limited compute resources that still need to equip models with up-to-date or specialized knowledge.
Technical Details
- KItCAT operates by stochastically corrupting input token sequences during continued pre-training: a random subset of tokens is replaced with arbitrary vocabulary tokens, while the original next-token prediction labels remain unchanged
- This corruption mechanism acts as a form of data augmentation, generating diverse training inputs from each original sample without requiring external paraphrasing models or additional LLM inference
- The method is applied to decoder-only LLM architectures using standard auto-regressive training objectives, making it broadly compatible with existing training pipelines
- Evaluations demonstrate consistent improvements over baseline CPT across multiple benchmark datasets and different model families, suggesting the approach generalizes well beyond specific architectures
- The computational overhead is described as negligible compared to paraphrase-based augmentation methods, as corruption requires no additional model calls or generation steps
Industry Insight
- Organizations performing domain adaptation on limited budgets can adopt KItCAT as a drop-in replacement for paraphrase-heavy knowledge injection pipelines, significantly reducing compute costs while maintaining or improving performance
- The simplicity of the approach suggests it could become a standard regularization technique during continued pre-training, not just for knowledge injection but potentially for improving robustness to input perturbations
- As niche and proprietary data becomes increasingly valuable for enterprise LLM deployments, lightweight training strategies like KItCAT will likely see broader adoption, especially in settings where access to powerful LLMs for paraphrase generation is restricted
Disclaimer: The above content is generated by AI and is for reference only.