Learnability-Informed Fine-Tuning of Diffusion Language Models

Background

Diffusion language models (DLMs) have gained attention for their ability to handle complex and nuanced text generation tasks. However, their reasoning capabilities are often suboptimal when compared to autoregressive models like transformers enhanced through supervised fine-tuning (SFT). Traditional SFT methods excel with common tokens but struggle with rare ones due to extensive input masking during training.

Key Points

1. Challenges of Vanilla SFT in DLMs: The analysis highlights that vanilla SFT overlooks learnability, particularly for rare tokens when most input is masked. Conversely, learning common tokens under unmasked conditions adds little value.
2. LIFT Algorithm: LIFT proposes a novel approach to align token learning with the available context at different diffusion time steps. It learns hard-to-learn tokens (rare and important ones) during masking phases and easy-to-learn tokens (common but less critical ones) when more context is available.
3. Experiments and Results: The algorithm was tested across six reasoning benchmarks, demonstrating superior performance over existing SFT baselines, with up to a 3x relative gain on AIME'24 and AIME'25.

Significance

1. Performance Improvement: LIFT significantly enhances the reasoning capabilities of DLMs by addressing the inherent limitations of vanilla SFT in learning rare tokens.
2. Practical Application: By aligning the training process with varying levels of context, LIFT can be applied to improve a wide range of natural language tasks that require deep understanding and reasoning.
3. Open Source Availability: The availability of code on GitHub facilitates further research and development, potentially leading to broader improvements in DLMs.

Key Insights:

• Understanding the learnability of tokens is crucial for effective training in DLMs.
• Contextual alignment during fine-tuning can significantly enhance model performance.
• Open-source contributions like LIFT promote collaborative advancements in NLP technology.