Memorization Dynamics of Fill-in-the-Middle Pretraining

Background

The article investigates the effect of Fill-in-the-middle (FIM) pretraining on the memorization dynamics of causal language models. Unlike traditional left-to-right (LTR) training objectives, FIM aims to equip models with infilling ability by filling in missing parts of sentences during pretraining. This study compares the memorization behaviors of FIM and LTR training through a controlled experiment using a matched pair of Llama 3.2 models.

Key Points

• Memorization Dynamics: The study finds that FIM more often recovers short or partially matching spans, whereas LTR assigns higher confidence to long exact continuations.
• Linear Relationship with Repetitions: Verbatim extraction under FIM training grows approximately linearly with the number of repetitions in the corpus tested. This relationship was observed across different lengths and probe formats.
• Prefix Context Importance: Evaluating native FIM-format probes showed that suffix context alone is insufficient; verbatim recall remains strongly anchored in prefix context.

Significance

The research highlights the nuanced differences in memorization strategies employed by models trained with FIM versus LTR objectives. Understanding these dynamics can provide insights into designing more effective pretraining methods and improve language model performance in various tasks. The findings also suggest that evaluating only one span length or probing format may overlook important aspects of the memorization behavior.

Key Insights:

• FIM vs. LTR Memorization: FIM promotes a preference for shorter, partially matching spans, while LTR favors longer exact continuations.
• Memorization Depth: The linear relationship between repetitions and verbatim extraction under FIM training indicates that increased exposure leads to proportionally greater memorization.
• Contextual Anchoring: Verbatim recall in FIM-trained models is heavily influenced by the prefix context, indicating a need for careful consideration of both prefix and suffix contexts when designing probes.

This analysis reveals the importance of understanding the specific memorization behaviors induced by different training objectives, which can inform future model development and evaluation strategies.