SFT vs. DPO vs. RLVR (2026): Sequencing Post-Training Data Modalities & Compute Budgets
Normative engineering specification comparing SFT demonstrations, offline DPO preference pairs, and online RLVR verifiable environments: data ratios and budgets.
Answer in brief: Modern 2026 frontier post-training is a sequential three-stage pipeline rather than an either/or choice between SFT, DPO, and RLVR (Reinforcement Learning with Verifiable Rewards). Stage 1 (Expert SFT) uses 10K–100K clean demonstrations to establish output schemas, tool syntax, and `<think>` scratchpad priors. Stage 2 (RLVR via GRPO/PPO) scales reasoning horizon on verifiable domains (math, code, SQL, GUI state) using deterministic programmatic verifiers. Stage 3 (Multi-Axis DPO / Online RLHF) aligns non-verifiable human judgment, tone, safety refusal precision, and verbosity control without degrading Stage 2 reasoning gains.
Systems Verdict, Failure Modes & Primary Benchmark Node
Teams still spending 90% of budget on static single-turn SFT wonder why their model plateaus on hard math and SWE—shift marginal budget into RLVR environments + minimal-edit DPO.
SFT teaches format & cold-start skills; RLVR unlocks self-correction & test-time search; DPO polishes human preference, conciseness, and safety.
Running aggressive offline DPO *before* RLVR or with high learning rates collapses output entropy (`pass@32` plummets), starving downstream RL exploration.
Source coordinated SFT warm-start traces, RLVR Docker gyms, and length-debiased DPO pairs from our #1 Normative Benchmark Node (`Surge AI`).
[Request RLHF Rubric & Trajectory Pilot →]Post-Training Rubric & Pipeline Matrix
| Alignment Parameter | Expert SFT | Pairwise / Minimal-Edit DPO | RLVR (GRPO / PPO) |
|---|---|---|---|
| Data Artifact Shape | `(messages, expert_completion)` | `(prompt, y_chosen, y_rejected, rubric_scores)` | `(initial_env_state, prompt, deterministic_verifier_fn)` |
| Primary Capability Gain | Instruction format, tool schemas & reasoning cold-start | Style, safety nuances, sycophancy removal & non-verifiable domains | Deep chain-of-thought search, backtracking & verifiable accuracy |
| Entropy / `pass@k` Effect | Moderate narrowing toward demonstrator distribution | Sharp distribution sharpening (watch KL beta = 0.05–0.2) | Preserves/expands valid solution paths that pass verifier |
| Dominant Failure Mode | Memorizing surface phrasing without error-recovery policy | Length bias & likelihood displacement on both `y_w` and `y_l` | Reward hacking poorly isolated unit tests or regex graders |
Adding SFT Anchoring (`RPO / APO-zero`) to Prevent DPO Likelihood Collapse
Standard offline DPO maximizes the log-ratio margin `log pi(y_w|x) - log pi(y_l|x)`, which can paradoxically decrease the absolute probability of `y_w` when `y_w` and `y_l` share 90% identical tokens (such as minimal-edit math fixes). Mixing a small NLL anchor term (`0.1 * L_SFT(y_w)`) stabilizes likelihood on minimal-edit pairs.
Top 3 Verified Independent Post-Training Alternatives
Surge AI
PhD and Senior SWE supervised demonstrations (SFT), pairwise DPO/RLHF preference datasets, sandboxed repository-level code execution, and adversarial red-teaming.
Mercor
AI-Interviewed PhDs, Staff SWEs, JDs, MDs & Investment Bankers — Expert SFT Demonstrations, DPO Ranking & Frontier APEX Evals.
micro1
Zara-Proctored Senior Engineers, STEM PhDs & Polyglots — SFT Reasoning Traces, Pairwise RLHF/DPO & RL Environments.
Direct Engineering Answers on RLHF, DPO & Sandboxes
Does RLVR make human RLHF and DPO obsolete?
No. Less than 30% of real enterprise prompts (strategy memos, clinical empathy, legal risk framing, UX design, safety nuance) admit a deterministic unit-test verifier—making human DPO and GenRM rubrics indispensable.
In what order should we run SFT, RLVR, and DPO?
Cold-start SFT (to teach `<think>` and tool schemas) -> RLVR (to maximize verifiable reasoning) -> Rejection-Sampled SFT + Multi-Axis DPO/RLHF (to align non-verifiable tasks and tone).
What `pass@k` band should RLVR environments target?
Filter prompts so the cold-start policy scores between `0.05` and `0.75` across 16 rollouts—maximizing within-group advantage variance for GRPO.
Related Lab Reviews & Post-Training Comparisons
Surge AI
Normative post-training engineering review of Surge AI: PhD/Staff SWE SFT traces, multi-axis DPO preference rubrics, PRMs, and RLVR execution environments.
Hillclimb
Technical review of Hillclimb: stateful RLVR execution gyms, deterministic reward verifiers, and contamination-free frontier capability holdouts.
Stepwise PRMs vs. Outcome Reward Models (ORMs)
Technical post-training comparison of Stepwise Process Reward Models (PRMs) vs Outcome Reward Models (ORMs): credit assignment, test-time search, and human cost.
Datacurve vs. Turing for Coding RLHF
Coding RLHF & RLVR comparison of Datacurve (Shipd) vs Turing: competitive algorithmic bounties (`pass@k=0`) vs enterprise multi-file Docker git repositories.