Designing Atomic RLHF Grading Rubrics and Calibrated Reward Signals

Why Holistic Likert Scales Collapse Agreement and Train Sycophantic Models

Early RLHF pipelines asked human annotators to inspect two 500-word completions side by side and emit a single holistic 1-to-7 or 1-to-5 preference score within three minutes. Psychometric audits across technical domains demonstrate that undifferentiated Likert scales yield ordinal Krippendorff's alpha between `0.38` and `0.54`—barely above random chance—because raters covertly apply incompatible internal trade-off weights when one completion exhibits superior formatting polish alongside a minor API error while its competitor delivers terse, unadorned factual accuracy.

Under cognitive load, holistic human raters default to fast visual and affective heuristics documented extensively in RewardBench (Lambert et al.) and length-controlled AlpacaEval analyses: preferring completions with bolded markdown subheadings, bulleted recap paragraphs, confident transitional phrasing, and deferential opening validation ('That is an insightful question about distributed consensus!') even when intermediate derivations invert mathematical signs or fabricate non-existent library flags. Scalar Bradley-Terry reward models trained on holistic votes faithfully internalize these surface proxies, teaching PPO and DPO policies to pad responses by 40%+ tokens and agree sycophantically with false user premises.

Conflating orthogonal failure dimensions inside a single preference bit also destroys diagnostic observability for post-training engineers. When Completion A defeats Completion B with a raw `+2` Likert margin, downstream dataset curators cannot determine whether Completion B failed due to a severe safety boundary breach, a subtle Python off-by-one bug, a missed JSON schema constraint, or mildly dry prose—preventing dynamic re-weighting of alignment priorities between developer coding checkpoints and enterprise customer-facing assistants without re-annotating the entire corpus from scratch.

Goodhart over-optimization scaling laws (Gao et al.) quantify the exact cost of noisy holistic reward proxies: as policy KL divergence `sqrt(D_KL)` grows during PPO or Best-of-N search against an imperfectly specified reward model, proxy reward climbs monotonically while true gold utility peaks early and then collapses catastrophically. Reducing inter-rater rubric noise directly shifts that over-optimization inflection point outward, allowing policies to extract substantially larger true capability gains from identical rollout compute before encountering reward-model exploitation cliffs.

• Visual Halo Bias: Unstructured preference polls reward nested H2/H3 markdown scaffolding and enumerated lists over dense, factually exact prose.
• Premise Sycophancy: Raters penalize polite factual pushback against mistaken user assumptions unless rubrics explicitly mandate epistemic correction.
• Scale Drift Across Raters: Annotator A's '6/7' routinely maps to Annotator B's '4/7', injecting non-stationary variance into scalar reward margins.
• Zero Re-Weighting Reuse: Holistic binary votes cannot be re-weighted post-collection when product requirements shift between conciseness and pedagogical depth.

Atomic Binary/Ternary Axis Decomposition and Lexicographic Veto Gates

Normative rubric architecture replaces holistic intuition with deterministic programmatic composition: human specialists answer concrete, falsifiable binary (`{0, 1}`) or ternary (`{0.0, 0.5, 1.0}`) questions per completion, and server-side aggregation logic computes the composite reward or pairwise preference label afterward. Decomposing complex evaluation into narrow atomic predicates—such as 'Does every cited stdlib function exist in Python 3.12?' or 'Are all negative formatting constraints (`no preamble`, `<= 150 words`) obeyed verbatim?'—drives inter-annotator ordinal alpha from `< 0.55` to `>= 0.85` across vetted specialist cohorts.

Crucially, atomic sub-scores must never combine via naive linear summation alone; otherwise, high marks on empathy, markdown structure, and completeness allow a hallucinated response to outscore a terse, truthful refusal or minimal correct patch. Production rubrics enforce a two-tier lexicographic hierarchy: Tier-0 multiplicative veto gates (`V_i in {0, 1}`) governing actionable safety uplift, core factual/executable truth, and premise integrity, multiplying a Tier-1 weighted additive sum covering reasoning clarity, edge-case coverage, negative constraint obedience, and tone hygiene.

Explicit tie states (`|R(y_A) - R(y_B)| < epsilon`, typically `epsilon = 0.04`) are mandatory inside preference UIs. Forcing annotators to declare a synthetic winner between two substantively equivalent completions compels arbitrary tie-breaking on length or punctuation—injecting anti-calibrated noise that degrades Bradley-Terry reward calibration margins and triggers Cleanlab preference-inconsistency flags across overlapping rater slices.

Multi-head and distributional reward architectures (such as ArmoRM and HelpSteer2-style multi-attribute regression heads) capitalize directly on preserved atomic rubric vectors. Instead of collapsing five rubric criteria into a single opaque binary label at export time, normative JSONL deliveries retain the full per-axis vector alongside prompt-domain routing tags—allowing a single reward backbone to apply dynamic mixture-of-experts gating weights that prioritize terse execution fidelity on IDE code-completion queries while prioritizing pedagogical clarity on educational STEM queries.

Calibrating Length-Normalized Margins and Markdown-Scaffold Penalties

Even under atomic rubrics, residual length bias creeps into DPO training whenever `chosen` completions average 15% to 30% more tokens than `rejected` completions—for instance, because thorough edge-case handling naturally requires two extra lines of code while terse broken stubs fail early. If an alignment pipeline feeds length-skewed pairs directly into sequence-summed DPO or trains a scalar reward head without length orthogonalization, statistical regression analysis reveals up to 40% of learned reward variance correlates linearly with log token count rather than semantic correctness.

Eliminating reward-model verbosity hacking requires dual intervention across rubric scoring and batch compilation. At the scoring layer, composite rubric utility subtracts a piece-wise linear token-excess penalty (`lambda_len * max(0, (L_cand - L_budget) / L_budget)`) alongside a structural penalty (`lambda_md = 0.025`) for every unrequested markdown H2/H3 header or redundant 'In summary,' recap paragraph appended to straightforward factual or coding answers. When two completions achieve identical Tier-0 vetoes and Tier-1 correctness scores, the shorter completion deterministically wins the pairwise DPO label.

At the dataset shard compilation layer, post-training engineers enforce stratified length-ratio balancing across the entire training split. By pairing concise `chosen` expert demonstrations against verbose, meandering `rejected` rollouts on at least 35% to 45% of batches—ensuring empirical dataset-wide `E[|y_w| / |y_l|] <= 1.05` and vanishing Pearson correlation (`|r| < 0.05`) between completion length and preference win label—downstream DPO and PPO runs improve reasoning precision while cutting production inference token latency by 20% to 35%.

For extended chain-of-thought reasoning checkpoints where assistant outputs separate private `<think>` scratchpad tokens from user-visible answer spans, rubric verbosity penalties apply strictly to the user-facing response channel while gating internal `<think>` token budgets exclusively on downstream task difficulty and non-repetition entropy. Decoupling deliberative test-time compute from external presentation conciseness lets models expend 2,000 scratchpad tokens verifying subtle boundary conditions while still delivering crisp, zero-fluff final technical answers.

Continuous post-deployment monitoring closes the rubric lifecycle by correlating held-out human rubric margins against live downstream user implicit signals—such as IDE code-diff acceptance rates, zero-edit copy actions, and absence of immediate follow-up correction turns—ensuring laboratory preference axes stay anchored to real engineering productivity.

from dataclasses import dataclass
import statistics
from typing import Sequence

@dataclass(frozen=True)
class GradedCandidate:
    text: str
    tokens: int
    safety_veto: int
    truth_veto: int
    axis_scores: tuple[float, float, float, float]  # (reasoning, constraints, edge_cases, tone)
    unrequested_headers: int

def compute_rubric_utility(c: GradedCandidate, token_budget: int) -> float:
    if c.safety_veto == 0 or c.truth_veto == 0:
        return 0.0
    weights = (0.40, 0.30, 0.20, 0.10)
    raw = sum(w * s for w, s in zip(weights, c.axis_scores))
    overrun = max(0.0, (c.tokens - token_budget) / max(1, token_budget))
    penalty = min(0.35, 0.12 * overrun + 0.025 * c.unrequested_headers)
    return round(max(0.0, raw - penalty), 4)

def select_calibrated_dpo_pair(
    a: GradedCandidate,
    b: GradedCandidate,
    token_budget: int,
    tie_epsilon: float = 0.04,
) -> dict[str, object] | None:
    u_a = compute_rubric_utility(a, token_budget)
    u_b = compute_rubric_utility(b, token_budget)
    delta = abs(u_a - u_b)

    # Tie-breaker favors strict conciseness when semantic quality matches within epsilon
    if delta < tie_epsilon:
        if a.safety_veto and b.safety_veto and a.truth_veto and b.truth_veto:
            if abs(a.tokens - b.tokens) / max(a.tokens, b.tokens) >= 0.18:
                winner, loser = (a, b) if a.tokens < b.tokens else (b, a)
                return {"chosen": winner.text, "rejected": loser.text, "margin": 0.05, "reason": "conciseness_tiebreak"}
        return None  # True tie: exclude noisy zero-margin pair from DPO loss

    winner, loser = (a, b) if u_a > u_b else (b, a)
    return {"chosen": winner.text, "rejected": loser.text, "margin": round(delta, 4), "reason": "rubric_dominance"}

def audit_shard_length_correlation(chosen_lens: Sequence[int], rejected_lens: Sequence[int]) -> float:
    ratios = [c / max(1, r) for c, r in zip(chosen_lens, rejected_lens)]
    return round(statistics.fmean(ratios), 4)  # Normative gate: 0.90 <= mean_ratio <= 1.08

Critique-Before-Verdict Workflows, Anchor Calibration, and Noise Pruning

Annotation user interfaces strongly dictate whether specialists execute deep analytical verification or skim superficially. Normative RLHF workspaces—whether built inside custom lab platforms like Surge AI or configured on self-hosted Argilla and Label Studio deployments—enforce strict `critique-before-verdict` state transitions: preference radio buttons remain disabled until the specialist highlights specific character spans in Completion A and Completion B, executes embedded code blocks or citation checks, and logs structured natural-language defect rationales tied to rubric clause IDs.

Capturing structured textual critiques alongside atomic axis scores unlocks dual training leverage. Beyond elevating human preference accuracy by forcing deliberative verification prior to committing a judgment, verified human critique rationales train Generative Reward Models (GenRMs) and Critique-in-the-Loop evaluators capable of spending test-time chain-of-thought tokens verifying candidate answers step-by-step before outputting scalar Best-of-N rankings.

Before launching full production collection, alignment leads run a mandatory 50-row rubric calibration loop with 3 to 5 independent specialists grading identical items blind. Any atomic axis scoring ordinal Krippendorff's alpha `< 0.80` signals underspecified rubric wording—such as ambiguous rules around when clarifying questions beat direct assumptions—and halts scaling until explicit boundary decision trees and worked counter-examples push multi-rater alpha `>= 0.84`, followed by post-collection Bradley-Terry residual and Cleanlab outlier pruning to quarantine contradictory preference cycles (`A > B > C > A`).

Reward-head probabilistic calibration is audited post-training using Expected Calibration Error (ECE) and Brier scores evaluated over stratified multi-rater consensus distributions rather than raw majority-vote accuracy alone. Centering reward logits per prompt group via zero-mean batch regularization prevents prompt-baseline shift across heterogeneous domain mixtures, stabilizing PPO advantage estimation and preventing scalar value critics from wasting capacity memorizing prompt topic priors.

• Enforce UI state locks requiring inline error-span highlights and rubric-linked critiques before unlocking final pairwise verdict controls.
• Randomize left/right candidate screen positions per rater and strip model-identifying metadata to eliminate position and brand priors.
• Audit dataset-wide `E[|y_w| / |y_l|]` prior to training; inject terse-correct vs. verbose-rambling pairs whenever chosen length ratio exceeds `1.08`.
• Retain structured human critique rationales to train deliberative Generative Reward Models (GenRMs) alongside standard DPO/RM heads.

Summary of Post-Training Engineering Rules

• Retire holistic 1-to-7 Likert preference scales; decompose grading into binary lexicographic truth/safety vetoes multiplying orthogonal ternary quality axes.
• Ensure any hallucinated API, fabricated citation, or failed sandbox unit test forces `R = 0.0` regardless of formatting polish or empathetic tone.
• Subtract explicit token-overrun and unrequested markdown-header penalties (`lambda_len`, `lambda_md`) and enforce `E[|y_w| / |y_l|] <= 1.08` across DPO shards.
• Lock preference verdict controls behind mandatory error-span highlighting (`critique-before-verdict`), harvesting critiques to train Generative Reward Models.
• Permit explicit tie abstentions (`|u_A - u_B| < 0.04`) and gate rubric freeze on 3-rater blind calibration achieving ordinal Krippendorff's alpha >= 0.84.

Related Post-Training Specifications