Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Introduces Procedural Memory Distillation (PMD), a method that captures cross-episode procedural signals often lost in standard RLVR and self-distillation approaches. Organizes memory into three abstraction levels: raw trajectories, self-reflected strategies, and recurring behavioral patterns, all extracted online. Implements a co-evolutionary loop where the policy updates the memory and the memory-conditioned self-teacher supervises the policy. Achieves significant performance gains over SDPO,
Analysis
TL;DR
- Introduces Procedural Memory Distillation (PMD), a method that captures cross-episode procedural signals often lost in standard RLVR and self-distillation approaches.
- Organizes memory into three abstraction levels: raw trajectories, self-reflected strategies, and recurring behavioral patterns, all extracted online.
- Implements a co-evolutionary loop where the policy updates the memory and the memory-conditioned self-teacher supervises the policy.
- Achieves significant performance gains over SDPO, improving by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH.
- Results in a memory-free model at inference time, as procedural knowledge is fully distilled into the policy weights.
Why It Matters
This approach addresses a critical limitation in current reinforcement learning frameworks for LLMs: the inability to retain and reuse rich procedural information across multiple training episodes. By converting transient rollout data into persistent, distilled knowledge, PMD enables models to learn from their own historical failures and successes more effectively, leading to substantial improvements in reasoning and coding benchmarks without increasing inference overhead.
Technical Details
- Methodology: PMD converts cross-episode signals into reusable procedural memory, which acts as a training scaffold. This memory is distilled into the policy's weights, allowing for a memory-free inference state.
- Memory Abstraction Levels: The memory structure operates at three levels: (1) raw trajectories, (2) self-reflected strategies and lessons, and (3) higher-level behavioral patterns that recur across different problems.
- Co-Evolution Mechanism: The core design involves a feedback loop where the policy generates rollouts that update the memory, while a memory-conditioned self-teacher uses this accumulated experience to supervise the student model during its own rollouts.
- Empirical Validation: Tested on Qwen3-8B and OLMo3-Instruct-7B, showing superior performance compared to SDPO. Ablation studies indicate that freezing either the memory or the policy component reduces performance by more than 10%, highlighting the necessity of the co-evolutionary process.
Industry Insight
- Efficiency in Training: The ability to distill complex procedural knowledge into static weights suggests a pathway to more efficient training loops that do not require maintaining large external memory buffers during inference, reducing deployment complexity.
- Focus on Process over Outcome: Moving beyond simple reward verification to capturing "how" solutions were derived (procedural memory) could become a standard practice for improving model robustness in complex reasoning tasks like coding and scientific evaluation.
- Importance of Co-Evolution: The significant drop in performance when decoupling memory and policy updates underscores the need for integrated training architectures where knowledge retention and policy optimization are mutually reinforcing rather than sequential steps.
Disclaimer: The above content is generated by AI and is for reference only.