SFT Expert Demonstrations vs. Pairwise Preference Optimization (DPO, KTO & SimPO)

Mode-Covering Behavior Cloning vs. Mode-Seeking Contrastive Gradients

Autoregressive Supervised Fine-Tuning minimizes assistant-masked token-level negative log-likelihood (`-sum log pi_theta(y_t | x, y_<t)`) directly over ground-truth human demonstrations. Because cross-entropy gradients pull probability density toward exact expert token trajectories regardless of the base checkpoint's prior likelihood, SFT acts as a mode-covering operator capable of implanting entirely unseen behaviors: structured JSON tool-invocation grammars, extended chain-of-thought scratchpad delimiters, multi-file unified diff syntax, and domain-specific proof idioms where a raw pre-trained checkpoint exhibits `pass@64 == 0`. Without an initial SFT warmstart establishing positive probability support over valid output schemas, downstream reinforcement learning or pairwise preference loops starve for informative positive rollouts.

Once an SFT checkpoint achieves non-trivial multi-sample support (`0.15 <= pass@16 <= 0.75`), however, continuing to scale human SFT volume encounters severe diminishing returns and eventual entropy degradation. Human specialists solving identical complex prompts employ heterogeneous architectural decompositions, variable naming conventions, and proof orderings; forcing a single parametric policy to maximize likelihood simultaneously across incompatible expert styles induces probability-mass smearing across high-temperature transitions while over-memorizing surface lexical idiosyncrasies. Empirically, over-training on static SFT shards beyond 2 to 3 epochs inflates validation perplexity on out-of-domain reasoning tasks and compresses `pass@32` exploration breadth critical for subsequent Reinforcement Learning with Verifiable Rewards (RLVR).

Pairwise preference optimization—canonicalized by Direct Preference Optimization (Rafailov et al.)—replaces token-cloning cross-entropy with a contrastive Bradley-Terry margin objective parameterized through implicit log-probability ratios against a frozen reference policy `pi_ref`. Rather than forcing the policy to reproduce one arbitrary human reference string verbatim, DPO increases the relative log-ratio `beta * log(pi_theta(y_w | x) / pi_ref(y_w | x))` over rejected counterparts `y_l` while permitting the model to retain its own fluent internal phrasing. Because human domain specialists evaluate and critique two pre-generated model completions 4x to 7x faster than authoring an error-free 1,200-token demonstration from scratch, preference pairs dominate marginal alignment ROI once baseline structural competence exists.

• Cold-Start SFT Mandate: Allocate proctored PhD and Staff-SWE SFT budgets strictly to capability regimes where base `pass@16 < 0.10` or novel tool/CoT wire schemas must be introduced.
• Contrastive Mode Sharpening: Transition annotation spend to pairwise DPO or unary KTO once SFT warmstarts establish `pass@16 >= 0.15`, suppressing subtle hallucinations and formatting defects.
• Assistant-Only Loss Masking: Zero out cross-entropy gradients (`loss_mask = 0`) over system instructions, user prompts, and tool stdout/stderr payloads so SFT capacity concentrates exclusively on decision tokens.
• Preserving Exploration Entropy: Monitor sequence-level Shannon entropy and `pass@32 / pass@1` ratios after every SFT milestone to halt behavior cloning before policy diversity collapses.

Mathematical Trade-Offs Across DPO, IPO, KTO, SimPO, and RPO

Standard DPO assumes human preference judgments obey a noiseless or well-calibrated Bradley-Terry probabilistic model where reward margins map through a sigmoid link function. When deterministic human raters exhibit near-100% separation on clear-cut factual or unit-test contrasts (`P(y_w > y_l) -> 1`), unregularized DPO pushes the implicit reward margin toward infinity regardless of `beta`, overfitting finite preference shards and eroding calibration. Identity Preference Optimization (IPO) resolves deterministic margin runaway by regressing log-likelihood ratios toward a fixed target margin `1 / (2 * beta)` under squared loss, preventing policy collapse on noiseless code and STEM contrast pairs.

Pairwise collection is not always operationally optimal when historical production logs contain isolated thumbs-up/thumbs-down telemetry or binary sandbox pass/fail outcomes lacking matched sibling completions on identical prompts. Kahneman-Tversky Optimization (KTO) reformulates alignment around prospect-theoretic human utility functions operating over unpaired unary examples `(x, y, label in {desirable, undesirable})` anchored against a batch-estimated KL baseline. While KTO unlocks low-cost unary binary signals and tolerates severe desirable-to-undesirable class imbalance (up to 1:9), controlled ablations on subtle multi-constraint coding and legal reasoning show paired on-policy DPO retains superior gradient signal-to-noise because identical prompt prefixes cancel prompt-difficulty variance.

A critical pathology shared by vanilla DPO and IPO is simultaneous likelihood squeezing: gradient updates can widen the log-ratio margin `log pi_theta(y_w) - log pi_theta(y_l)` while driving down the absolute log-probability of `chosen` sequences `pi_theta(y_w | x)`, shifting probability mass onto out-of-distribution degenerate repetitions or verbose token sequences. Reference-free Simple Preference Optimization (SimPO) normalizes log-likelihoods strictly by sequence token count (`(beta / |y|) * log pi_theta(y | x)`) with a hard target margin `gamma`, eliminating both reference-model GPU memory overhead and length-exploitation incentives, while Regularized Preference Optimization (RPO / DPO+NLL) adds a small `alpha * NLL(y_w)` anchor term (`alpha in [0.05, 0.20]`) that prevents `chosen` likelihood decay during reasoning alignment.

Why Off-Policy Preference Corpora Fail Without On-Policy Rollout Refresh

The single costliest mistake in enterprise post-training procurement is commissioning 50,000 static 'off-policy' DPO pairs sampled from third-party commercial APIs rather than generating candidate completions directly from the buyer's active SFT checkpoint. Under off-policy DPO, both `y_w` and `y_l` occupy low-probability regions under `pi_ref`; gradient updates evaluate the policy's decision surface far from trajectories the model actually emits during greedy or temperature-sampled inference. Consequently, off-policy DPO models score high accuracy on held-out static classification splits while continuing to emit unpenalized native hallucinations during live autoregressive generation.

Normative post-training pipelines enforce iterative `on-policy` or `semi-on-policy` preference collection. For every prompt `x` in the target frontier distribution, the orchestration harness samples `G = 4` to `8` completions `(y_1, ..., y_G)` directly from the current SFT or intermediate checkpoint across stratified temperatures (`T in {0.6, 0.9}`), filters exact duplicates, and routes high-disagreement pairs to proctored human specialists (such as Surge AI, Mercor, micro1, or Turing pods) for atomic rubric adjudication. When `pass@G == 0` on a hard prompt, a senior specialist authors or minimally edits a surgical `chosen` patch starting from the model's closest near-miss rollout—minimizing edit distance so `y_w` remains close to the policy manifold.

Controlling sequence-level Kullback-Leibler divergence (`D_KL(pi_theta || pi_ref)`) across iterative DPO rounds requires precomputing exact per-token reference log-probabilities prior to training and monitoring empirical forward-KL budgets continuously. Whenever mean sequence KL exceeds `12` to `18` nats on 1,024-token completions without proportional gains on deterministic execution benchmarks, alignment engineers refresh `pi_ref <- pi_theta`, regenerate fresh on-policy candidate pairs against updated failure modes, and discard stale historical preference shards whose `rejected` errors the checkpoint has already extinguished during earlier alignment iterations.

import torch
import torch.nn.functional as F

def compute_rpo_dpo_loss(
    policy_chosen_logps: torch.Tensor,     # shape: (batch,) summed over assistant mask
    policy_rejected_logps: torch.Tensor,   # shape: (batch,)
    ref_chosen_logps: torch.Tensor,        # shape: (batch,) precomputed from frozen SFT
    ref_rejected_logps: torch.Tensor,      # shape: (batch,)
    chosen_token_counts: torch.Tensor,     # shape: (batch,)
    rejected_token_counts: torch.Tensor,   # shape: (batch,)
    beta: float = 0.10,
    nll_alpha: float = 0.10,
    length_reg_gamma: float = 0.02,
) -> tuple[torch.Tensor, dict[str, float]]:
    chosen_logratios = policy_chosen_logps - ref_chosen_logps
    rejected_logratios = policy_rejected_logps - ref_rejected_logps

    # Explicit length-difference regularizer prevents exploiting positive log-ratio drift via verbosity
    length_delta = (chosen_token_counts - rejected_token_counts).float() / chosen_token_counts.clamp(min=1)
    logits = beta * (chosen_logratios - rejected_logratios) - (length_reg_gamma * length_delta)

    dpo_loss = -F.logsigmoid(logits)
    # RPO anchor prevents simultaneous likelihood collapse on verified chosen reasoning traces
    nll_anchor = -policy_chosen_logps / chosen_token_counts.clamp(min=1).float()
    total_loss = (dpo_loss + nll_alpha * nll_anchor).mean()

    with torch.no_grad():
        reward_acc = (chosen_logratios > rejected_logratios).float().mean().item()
        mean_margin = (beta * (chosen_logratios - rejected_logratios)).mean().item()

    return total_loss, {"reward_accuracy": reward_acc, "implicit_margin": mean_margin}

Sequencing SFT-to-DPO Ratios and Minimal-Edit Chosen Surgery

Across 7B-to-400B parameter post-training runs, optimal annotation capital allocation follows an inverted funnel: fewer, ultra-pure SFT demonstrations paired with broader, iteratively refreshed on-policy preference pairs. Rather than diluting SFT shards with 200,000 mediocre crowd-written answers—a practice disproven by LIMA and subsequent frontier scaling studies—high-performing post-training curricula curate 5,000 to 25,000 rigorously proctored PhD and Staff-SWE demonstrations stratified via dense embedding clustering (Lilac / HNSW k-center greedy) to maximize task-manifold coverage with zero redundant boilerplate.

When human intervention is needed inside preference pipelines because every sampled candidate `(y_1, ..., y_G)` contains a subtle trailing defect, 'minimal-edit surgical revision' vastly outperforms full human rewrites as the DPO `chosen` target. If a Staff SWE rewrites an entire 80-line module from scratch in their personal coding style while `rejected` reflects the model's native token distribution, the DPO classifier trivially separates `chosen` from `rejected` via stylistic surface features rather than learning the 3-line concurrency bug fix. Constraining specialists to apply the minimum necessary diff to transform a sampled model rollout into a passing `chosen` completion isolates causal gradient credit directly onto the corrected defect tokens.

Multi-stage post-training schedules interleave SFT, RLVR, and DPO in disciplined sequence: Stage 1 trains an SFT warmstart preserving wide output entropy (`T = 1.0` calibration); Stage 2 runs online RLVR (GRPO/PPO) over verifiable math, code, and tool-use gyms where deterministic oracles exist; and Stage 3 applies length-balanced RPO/DPO across mixed verifiable and open-ended preference shards (`beta = 0.08` to `0.15`) to align conversational pragmatics, tone neutrality, and refusal precision without regressing hard reasoning `pass@1` scores.

Gradient packing and sequence-length stratification during SFT warmstarts also materially impact downstream preference stability. Naively concatenating short 120-token chat turns alongside 16,384-token repository trajectories inside unmasked FlashAttention sequences without cross-document attention resets (`cu_seqlens` boundary isolation) leaks cross-example context representations and skews per-batch denominator weighting toward verbose outliers. Normalizing SFT cross-entropy per valid assistant token across length-homogeneous micro-batches ensures concise analytical demonstrations exert proportional gradient magnitude alongside multi-thousand-token reasoning chains.

• Curate 5k–25k high-entropy, proctored SFT demonstrations deduplicated via semantic k-center clustering rather than 100k+ low-rigor crowd examples.
• Generate DPO candidates strictly on-policy (`G = 4–8` rollouts from the active checkpoint) and refresh `pi_ref` whenever sequence KL exceeds ~15 nats.
• Prefer minimal-edit surgical human patches on near-miss model rollouts over full stylistic rewrites when constructing `chosen` DPO targets.
• Include an `alpha = 0.05–0.15` per-token NLL anchor (RPO) on reasoning-heavy DPO runs to block simultaneous `chosen`/`rejected` log-probability decay.

Summary of Post-Training Engineering Rules

• Use proctored PhD and Staff-SWE SFT demonstrations strictly as a mode-covering warmstart to implant novel schemas, tool syntax, and non-zero `pass@16` support.
• Pivot post-training budgets to pairwise on-policy DPO, SimPO, or RPO once `pass@16 >= 0.15`, capturing 4x–7x higher human evaluation throughput per dollar.
• Avoid static off-policy third-party DPO datasets; sample candidate pairs directly from the active checkpoint and anchor reasoning pairs with an RPO NLL term.
• Construct `chosen` fixes via minimal-edit surgery on near-miss model rollouts so contrastive gradients isolate causal bug fixes rather than stylistic idiolects.
• Monitor `chosen` vs. `rejected` token-length parity (`0.85 <= ratio <= 1.20`) and sequence KL divergence continuously to prevent verbosity reward hacking.

Related Post-Training Specifications