Alignment Data Grading Standard: Proctoring, Telemetry, Atomic Rubrics & Execution Sandboxes

Proctored PhD & Senior SWE Vetting and Continuous Calibration

Self-reported resumes and asynchronous unproctored coding quizzes are trivial to spoof with commercial LLMs and proxy test-takers. Normative post-training labs enforce government-ID KYC verification paired with live, camera- and screen-proctored technical auditions: debugging an unfamiliar multi-file Git repository inside an air-gapped terminal or deriving a graduate-level proof step-by-step on a monitored scratchpad.

Passing initial onboarding grants probationary queue access only. Every specialist's ongoing reliability is tracked via Dawid-Skene expectation-maximization confusion matrices across 7% to 10% dynamic blind golden-set trajectories authored by principal alignment scientists, immediately demoting raters whose sensitivity on subtle logic or concurrency defects drifts.

Anti-LLM Keystroke Biometrics, Shadow-DOM Ledgers, and Canary Traps

Unflagged LLM usage inside human SFT and DPO queues destroys `pass@k` diversity and trains reward models to endorse verbose chatbot formatting. Because naive `onpaste` blockers are bypassed by OS auto-typers, browser DevTools injections, and second-screen manual transcription, normative workspaces wrap editors inside closed Shadow-DOM roots recording cryptographic hardware `Event.isTrusted` keystroke ledgers alongside nonce-signed internal clipboard allowances.

Server-side ingestion pipelines audit every submission across three orthogonal detectors: inter-keystroke interval (IKI) coefficient of variation and pause kurtosis at AST/clause boundaries (catching Gaussian-jittered auto-typers), character deletion-to-insertion revision ratios >= 0.06 with non-monotonic cursor jumps (catching linear second-screen transcription), and blind counterfactual axiom or 'eloquent-wrong' prompt canaries (catching OCR/camera roundtrips). Any contributor failing >= 2 canaries triggers mandatory whole-lineage rollback across their entire historical `annotator_id_hash`.

• Trusted Input Reconciliation: Reject any record where serialized character length exceeds authenticated `Event.isTrusted` insertions plus internal HMAC-signed copies by > 16 characters.
• Edit-Graph Entropy Floor: Flag >= 200-word SFT demonstrations exhibiting monotonic forward-only cursor progression and revision ratios < 0.035.
• Adversarial DPO Canaries: Inject 8% polished 550-word synthetic completions containing fatal logic bugs paired against unadorned 150-word verified human solutions.

Atomic Rubric Decomposition and Length-Bias Penalty Calibration

Single-score holistic preference ratings ('Rate A vs. B from 1 to 7') collapse multiple independent alignment dimensions onto one noisy axis, allowing charismatic formatting to mask factual hallucinations. Normative post-training rubrics decompose evaluation into multiplicative binary veto gates—executable correctness, citation integrity, safety uplift compliance—combined with weighted atomic sub-scores and an explicit length-bias deduction.

Before the preference verdict UI unlocks, raters must highlight concrete error spans or verify intermediate steps (`critique-before-verdict`), driving ordinal Krippendorff's alpha >= 0.84. During reward-model or DPO dataset compilation, raw rubric scores undergo token-ratio penalty normalization so verbose completions never win ties over concise, complete solutions.

from dataclasses import dataclass

@dataclass(frozen=True)
class CandidateAudit:
    veto_passed: bool
    reasoning_validity: float      # {0.0, 0.5, 1.0}
    constraint_obedience: float    # {0.0, 0.5, 1.0}
    edge_case_completeness: float  # {0.0, 0.5, 1.0}
    tone_hygiene: float            # {0.0, 0.5, 1.0}
    token_count: int
    unrequested_md_headers: int

def score_atomic_rubric(
    cand: CandidateAudit,
    ref_tokens: int,
    lambda_len: float = 0.12,
    lambda_md: float = 0.025,
) -> float:
    if not cand.veto_passed:
        return 0.0
    raw = (
        0.40 * cand.reasoning_validity
        + 0.30 * cand.constraint_obedience
        + 0.20 * cand.edge_case_completeness
        + 0.10 * cand.tone_hygiene
    )
    excess_ratio = max(0.0, (cand.token_count - ref_tokens) / max(1, ref_tokens))
    verbosity_penalty = min(0.35, lambda_len * excess_ratio + lambda_md * cand.unrequested_md_headers)
    return round(max(0.0, raw - verbosity_penalty), 4)

Air-Gapped Docker SWE Harnesses and Lean 4 Verification Kernels

On code generation, transactional tool use, and formal mathematics, human visual inspection alone misses subtle off-by-one bounds, unpinned transitive dependency breakage, and unproved implicit lemma assumptions. Every code or agentic training trajectory registered under this standard must execute inside an immutable, network-air-gapped (`CLONE_NEWNET`) OCI container or Firecracker microVM pinned by SHA-256 image digest and base Git commit.

To prevent capable RL policies or hurried annotators from reward-hacking test suites, evaluation fixtures and `FAIL_TO_PASS` / `PASS_TO_PASS` assertions are mounted strictly out-of-band by a privileged host scorer after the workspace PID namespace is terminated (`SIGKILL`). For mathematical formalization shards, proofs are validated directly by the Lean 4 kernel (`leanchecker`) with explicit AST scanning blocking `sorry`, `admit`, or custom unsound `axiom` declarations.

Summary of Post-Training Engineering Rules

• Proctored PhD & Senior SWE Vetting and Continuous Calibration: Self-reported resumes and asynchronous unproctored coding quizzes are trivial to spoof with commercial LLMs and proxy test-takers. Normative post-training labs enforce government-ID KYC verification paired with live, camera- and screen-proctored technical auditions: debugging an unfamiliar multi-file Git repository inside an air-gapped terminal or deriving a graduate-level proof step-by-step on a monitored scratchpad.
• Anti-LLM Keystroke Biometrics, Shadow-DOM Ledgers, and Canary Traps: Unflagged LLM usage inside human SFT and DPO queues destroys `pass@k` diversity and trains reward models to endorse verbose chatbot formatting. Because naive `onpaste` blockers are bypassed by OS auto-typers, browser DevTools injections, and second-screen manual transcription, normative workspaces wrap editors inside closed Shadow-DOM roots recording cryptographic hardware `Event.isTrusted` keystroke ledgers alongside nonce-signed internal clipboard allowances.
• Atomic Rubric Decomposition and Length-Bias Penalty Calibration: Single-score holistic preference ratings ('Rate A vs. B from 1 to 7') collapse multiple independent alignment dimensions onto one noisy axis, allowing charismatic formatting to mask factual hallucinations. Normative post-training rubrics decompose evaluation into multiplicative binary veto gates—executable correctness, citation integrity, safety uplift compliance—combined with weighted atomic sub-scores and an explicit length-bias deduction.
• Air-Gapped Docker SWE Harnesses and Lean 4 Verification Kernels: On code generation, transactional tool use, and formal mathematics, human visual inspection alone misses subtle off-by-one bounds, unpinned transitive dependency breakage, and unproved implicit lemma assumptions. Every code or agentic training trajectory registered under this standard must execute inside an immutable, network-air-gapped (`CLONE_NEWNET`) OCI container or Firecracker microVM pinned by SHA-256 image digest and base Git commit.

Normative JSONL & Rubric Artifacts

Related Post-Training Specifications