SFT, RL and DPO: The Other Stack
Post-training (SFT, DPO, PPO, GRPO, RLVR) is the critical stage that transforms pre-trained base models into usable assistants, and it is where most practical model behavior is determined LoRA and its variants (QLoRA) are the dominant parameter-efficient fine-tuning methods, freezing base weights to drastically reduce training memory while producing portable adapters DPO has largely replaced RLHF by eliminating the need for a reward model and RL loop, optimizing a single closed-form objective ag
Analysis
TL;DR
- Post-training (SFT, DPO, PPO, GRPO, RLVR) is the critical stage that transforms pre-trained base models into usable assistants, and it is where most practical model behavior is determined
- LoRA and its variants (QLoRA) are the dominant parameter-efficient fine-tuning methods, freezing base weights to drastically reduce training memory while producing portable adapters
- DPO has largely replaced RLHF by eliminating the need for a reward model and RL loop, optimizing a single closed-form objective against a frozen reference model
- GRPO with verifiable rewards (RLVR) is the leading approach for reasoning tasks, as demonstrated by DeepSeek-R1, and avoids the critic network overhead of classical PPO
- Reinforcement learning rollouts are fundamentally inference workloads, meaning serving stack performance (KV cache management, quantization, batching) directly determines post-training run duration
Why It Matters
Post-training is where the majority of a model's useful behavior is encoded, yet it remains poorly understood by practitioners who conflate it with serving. The article bridges a critical gap by showing that RL-based post-training is inseparable from the serving stack—rollout generation consumes the same KV cache, memory bandwidth, and quantization tradeoffs that serving engineers manage daily. For AI practitioners, this means investing in serving infrastructure optimization pays direct dividends in training throughput, and choosing the right post-training method can eliminate entire categories of model components (reward models, critics) without sacrificing performance.
Technical Details
- SFT (Supervised Fine-Tuning): Trains on curated prompt-response pairs using the same next-token prediction objective as pre-training. LoRA freezes base weights and trains small low-rank adapter matrices, reducing trainable parameters while keeping optimizer state minimal. QLoRA extends this by quantizing the frozen base to 4-bit, enabling large-model fine-tuning on consumer hardware.
- DPO (Direct Preference Optimization): Optimizes a single classification loss using prompt-better-answer-worse-answer triples, comparing the trained model against a frozen reference copy. Eliminates the reward model and RL loop required by RLHF, operating under the Bradley-Terry model assumption for preferences.
- GRPO (Group Relative Policy Optimization): Removes the critic network from PPO by sampling a group of G answers per prompt, scoring them, and using the group mean as the baseline for advantage computation. Reduces resident networks from four (PPO) to two (with verifier) or three (with learned reward model).
- RLVR (Reinforcement Learning with Verifiable Rewards): Replaces learned reward models with programmatic verifiers (answer keys, unit tests, compilers) that produce binary 0/1 rewards. Prevents reward model drift but introduces exploitation risks where models find valid answers through invalid routes.
- Serving-Training Connection: GRPO rollouts generate G answers per prompt for every batch item, creating serving-like workloads that hit the same KV cache and memory bandwidth limits. Quantization in rollouts carries dual costs: quality degradation (as in serving) plus broken reference-model comparison (unique to training).
Industry Insight
- The SFT → DPO → GRPO/RLVR decision tree should guide method selection: start with SFT for format/tone/domain issues, add DPO when ranking is possible but ideal answers cannot be written, and only reach RLVR when correctness is programmatically verifiable. PPO should be considered last and rarely, given its four-network overhead.
- Post-training infrastructure and serving infrastructure are now the same infrastructure. Teams that optimize their serving stack (KV cache efficiency, quantization, batching) directly reduce RL training costs and iteration cycles, making the two engineering functions more interdependent than ever.
- The rise of RLVR and open reasoning models (DeepSeek-R1 lineage) signals a shift away from human-labeled preference data toward programmatic verification, reducing annotation costs while introducing new failure modes around reward hacking that require rigorous checker design rather than model-scale investment.
Disclaimer: The above content is generated by AI and is for reference only.