Finetuning a Reasoning LLM with Supervised or Reinforcement Learning?
Supervised Fine-Tuning (SFT) is the recommended starting point for reasoning LLMs when high-quality, correct trajectories are available, rather than immediately resorting to complex reinforcement learning. Internal annotation formats (e.g., `assistant_think`, `assistant_tool`) must be strictly separated from training formats; data must be converted into the target model's specific chat template and tool-calling schema before training. Loss calculation should be selective; typically, loss is appl
Analysis
TL;DR
- Supervised Fine-Tuning (SFT) is the recommended starting point for reasoning LLMs when high-quality, correct trajectories are available, rather than immediately resorting to complex reinforcement learning.
- Internal annotation formats (e.g.,
assistant_think,assistant_tool) must be strictly separated from training formats; data must be converted into the target model's specific chat template and tool-calling schema before training. - Loss calculation should be selective; typically, loss is applied only to assistant responses, with reasoning spans optionally supervised depending on whether visible chain-of-thought is desired.
- Preference optimization (DPO) is viable if good/bad trajectory pairs can be constructed, while Reinforcement Learning (GRPO/RL) is reserved for scenarios with executable tools, rollout environments, and reliable reward signals.
Why It Matters
This guidance addresses a critical bottleneck in deploying advanced reasoning models: the misalignment between raw annotation data and model-specific training requirements. By clarifying the distinction between internal data structures and the actual chat templates used by frameworks like Hugging Face TRL or vLLM, practitioners can avoid subtle training failures caused by format mismatches. It provides a pragmatic roadmap for choosing between SFT, DPO, and RL based on available infrastructure and data quality, preventing wasted compute on inappropriate optimization methods.
Technical Details
- Data Representation Strategy: Raw annotations like
assistant_thinkandassistant_toolshould be treated as intermediate processing steps. Before training, these must be mapped to the target model's native format, such as standardassistantmessages containing reasoning text,tool_callsobjects, andtoolrole messages for results. - Tool-Calling Integration: Utilize libraries like TRL’s
SFTTrainerwhich support tool-calling natively. Datasets must include atoolscolumn with JSON schemas, ensuring the tokenizer and chat template correctly render function calls and their outputs. - Loss Masking Protocols: Do not apply loss to conditioning context (system prompts, user messages). For assistant responses, loss is generally applied to the final answer. Supervising the "thinking" or reasoning span is optional and depends on whether the goal is to make the model's internal reasoning process visible and reproducible.
- Template Specificity: Different model families (e.g., Qwen, GPT-OSS) require specific chat templates and parsers. Using generic roles without a corresponding custom chat template that renders them correctly will lead to poor performance, as the model expects specific control tokens and structures it was pre-trained on.
Industry Insight
- Standardize Data Pipelines Early: Adopt a normalization layer (similar to the Agent Data Protocol concept) that converts heterogeneous raw trajectories into a unified, model-specific schema before any training begins. This reduces friction when switching between base models or updating inference stacks.
- Evaluate Infrastructure Readiness for RL: Organizations should not jump to Reinforcement Learning for reasoning tasks unless they have robust execution environments and verifiable reward functions. For most cases, high-quality SFT combined with DPO offers a better return on investment and stability.
- Prioritize Chat Template Fidelity: Treat the chat template as a core component of the model interface, not just a formatting utility. Ensure that preprocessing pipelines strictly adhere to the template requirements of the target model family to maintain compatibility with serving engines like vLLM.
Disclaimer: The above content is generated by AI and is for reference only.