From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
GRIN (Golden-GRPO Injection) is a three-stage self-learning framework that enables continual knowledge injection in LLMs beyond simple memorization Golden-GRPO, a mixed-policy RL algorithm, injects golden answers to provide learning signals even when on-policy rollouts fail on novel facts Two new document-level benchmarks, Blank and Counter, evaluate novel acquisition and counterfactual overwrite across single-fact recall, multi-source retrieval, and inferential reasoning Mixed-policy reinforcem
Analysis
TL;DR
- GRIN (Golden-GRPO Injection) is a three-stage self-learning framework that enables continual knowledge injection in LLMs beyond simple memorization
- Golden-GRPO, a mixed-policy RL algorithm, injects golden answers to provide learning signals even when on-policy rollouts fail on novel facts
- Two new document-level benchmarks, Blank and Counter, evaluate novel acquisition and counterfactual overwrite across single-fact recall, multi-source retrieval, and inferential reasoning
- Mixed-policy reinforcement learning substantially outperforms SFT and mixed-policy RL baselines on harder question types while matching them on basic fact recall
- The work establishes that RL-based knowledge injection enables genuine "absorption" rather than superficial memorization of injected facts
Why It Matters
Continual knowledge injection is a critical bottleneck for keeping LLMs current without full retraining, and existing SFT-based approaches fundamentally fail at generalization across paraphrases and reasoning tasks. This work provides a practical RL-based alternative that could become the new standard for model updating pipelines, especially in domains requiring frequent knowledge refreshes.
Technical Details
- Golden-GRPO: A mixed-policy reinforcement learning algorithm where a golden answer is injected as a learning signal during training, allowing the model to learn from correct responses even when its own on-policy rollouts produce incorrect answers on novel facts
- Three-stage self-learning framework: GRIN structures the knowledge injection process into sequential stages that progressively build from basic recall to complex reasoning over injected knowledge
- Blank benchmark: A document-level evaluation targeting novel knowledge acquisition, testing single-fact recall, multi-source retrieval, and inferential reasoning
- Counter benchmark: A document-level evaluation targeting counterfactual overwrite, measuring the model's ability to replace outdated or incorrect knowledge with new information
- Both benchmarks evaluate performance across three difficulty tiers: single-fact recall, multi-source retrieval, and inferential reasoning, with GRIN showing the largest margins on the harder tiers
Industry Insight
- Organizations maintaining large language models should consider shifting from SFT-based updating pipelines to mixed-policy RL approaches, particularly for knowledge domains requiring reasoning over injected facts rather than simple recall
- The Blank and Counter benchmarks provide actionable evaluation frameworks for measuring whether knowledge updates genuinely generalize or merely memorize, which is critical for production model maintenance
- As model update frequency increases in fast-evolving domains, the gap between memorization-based and absorption-based approaches will widen, making RL-based injection methods increasingly essential for competitive model maintenance
Disclaimer: The above content is generated by AI and is for reference only.