Post-Training: The $100M Question Nobody's Asking About AI
Post-training is the critical phase that transforms a knowledge-rich but unusable base model into a helpful assistant, running through three stages: SFT, RLHF, and RLVR Quality of instruction data dramatically outweighs quantity, as demonstrated by LIMA (1,000 curated examples outperforming models trained on 52x more data) RLHF uses human preference feedback (via reward models) to teach models not just correctness but desirability of responses; DPO offers a simpler alternative RLVR leverages ver
Analysis
TL;DR
- Post-training is the critical phase that transforms a knowledge-rich but unusable base model into a helpful assistant, running through three stages: SFT, RLHF, and RLVR
- Quality of instruction data dramatically outweighs quantity, as demonstrated by LIMA (1,000 curated examples outperforming models trained on 52x more data)
- RLHF uses human preference feedback (via reward models) to teach models not just correctness but desirability of responses; DPO offers a simpler alternative
- RLVR leverages verifiable rewards (math, code) to let models discover reasoning strategies like chain-of-thought through practice rather than imitation
- Post-training is where a model's behavior, alignment, and character are determined — making it the most valuable and underappreciated lever in AI development
Why It Matters
Post-training is the affordable, high-impact differentiator in AI: while building foundation models costs hundreds of millions, fine-tuning an open model is comparatively cheap and determines how the model actually behaves in practice. For AI practitioners, this means proprietary data — good examples of work, user approvals, domain expertise — is far more valuable than raw compute, and represents the primary path to building competitive, specialized AI systems.
Technical Details
- SFT (Supervised Fine-Tuning): The model is trained on question-answer pairs with high-quality demonstrations, learning the format and patterns of helpful responses. The LIMA paper showed that 1,000 carefully curated examples on a 65B-parameter model outperformed a competitor trained on 52,000 examples, proving quality over quantity.
- RLHF (Reinforcement Learning from Human Feedback): A reward model is trained on human preference comparisons between candidate responses, then used to fine-tune the base model via reinforcement learning. The seminal result showed a 1.3B-parameter model with RLHF producing preferred answers over a 175B-parameter model without it. DPO (Direct Preference Optimization) is noted as a simpler, reward-model-free alternative now widely adopted by open-source teams.
- RLVR (Reinforcement Learning with Verifiable Rewards): Uses objective, computer-gradable rewards (math correctness, code test passing) instead of human feedback. This stage enabled the emergence of chain-of-thought reasoning — the model discovered that step-by-step thinking yields more correct answers, a strategy never explicitly taught.
- Key failure modes identified: Reward hacking (model optimizes for the reward signal rather than the actual task), sycophancy (model learns to agree with users even when wrong), and catastrophic forgetting (over-fine-tuning on one domain erases general capabilities).
Industry Insight
- The scarcity in AI is shifting from compute and data scale to high-quality human-generated examples and preference signals — companies should treat their internal documentation, approved outputs, and user feedback as strategic assets rather than exhaust.
- Post-training remains largely manual and subjective (annotator-dependent, no settled answer on whose preferences to encode), creating both a competitive moat for early movers and significant unresolved alignment risk that will define model behavior in production.
- The open-source ecosystem's adoption of DPO over full RLHF signals a trend toward simpler, more accessible alignment pipelines — lowering the barrier to entry for specialized model development while potentially sacrificing some nuance in preference learning.
Disclaimer: The above content is generated by AI and is for reference only.