Preventing LLM-Assisted Cheating in Human RLHF Annotation

How Unproctored Crowdworker LLM Usage Induces Reward Collapse

Whenever post-training pipelines source human demonstrations or preference rankings from unproctored crowd marketplaces compensated per completed task or loosely tracked billable hour, rational economic incentives push contributors to delegate cognitive effort to commercial LLMs. Annotators copy multi-constraint coding, STEM, or legal prompts into external assistant tabs, rewrite the opening clause or rename two local variables to defeat naive string detectors, and submit synthetic completions billed at senior specialist rates while clearing complex 45-minute domain reasoning tasks in under four minutes with near-zero genuine cognitive engagement.

Ingesting human SFT batches silently contaminated with 20% to 50% unverified commercial assistant outputs induces recursive model autophagy and tail-distribution collapse during fine-tuning. Rather than expanding the candidate model's reasoning horizon with high-entropy human domain solutions, contaminated shards pull token probabilities toward the homogenized stylistic priors of existing chatbots—inflating markdown subheading scaffolding, unsolicited transitional summaries, and polite hedging caveats while crushing multi-sample solution diversity (`pass@32` vs. `pass@1` scaling slopes) across downstream reinforcement learning rollouts, automated unit-test synthesis suites, and held-out frontier reasoning evaluations across production checkpoints.

Synthetic contamination inside Direct Preference Optimization (DPO) and scalar reward-model queues corrupts alignment gradients even more severely than tainted SFT demonstrations. Crowdworkers piping side-by-side candidate completions into an external LLM judge systematically endorse whichever response exhibits stronger stylistic homology, list density, and verbosity alignment with that helper model—inverting ground-truth preference labels on hard clinical, mathematical, or systems-engineering edge cases where terse factual precision contradicts confident synthetic hallucinations and training reward models that actively penalize concise, non-sycophantic expert truth during PPO, GRPO, or Best-of-N selection loops.

• Collapse of `pass@32` vs. `pass@1` scaling slopes caused by hidden synthetic homogenization across supposedly independent human demonstrations.
• Spurious reward-model length and markdown-header bias driven by LLM-generated preference rationales rubber-stamping verbose completions.
• Silent benchmark and pretraining leakage when external commercial assistants regurgitate memorized evaluation solutions into SFT training shards.
• Severely depressed inter-annotator variance masking low genuine comprehension until blind proctored holdout audits fail.

Keystroke Inter-Arrival Distributions, Edit-Graph Entropy, and Anti-Auto-Typer Detection

Disabling browser `paste` listeners on HTML textareas provides negligible operational security because organized annotation fraud rings immediately deploy OS-level auto-typer macros, browser DevTools state setters, programmable USB HID keyboard emulators, or dual-monitor manual re-typing. Reliable anti-synthetic verification requires recording full high-resolution input telemetry across every session: keystroke dwell durations (`keyup - keydown`), flight latencies (inter-keystroke intervals, IKI), non-adjacent cursor navigation offsets, and character-level insertion-deletion edit graphs cryptographically bound to the authenticated contributor session token from initial prompt render through final signed JSONL batch submission and server verification.

Genuine human authoring of complex analytical prose or executable code produces a distinct, non-stationary cognitive signature: heavy-tailed log-normal IKI distributions interrupted by 1.5-to-12-second planning pauses at clause transitions, mathematical derivations, and AST block boundaries, accompanied by non-linear cursor backtracking and a deletion-to-insertion revision ratio consistently falling between `0.08` and `0.26`. Scripted OS auto-typers—even variants injecting randomized Gaussian or uniform delay jitter—betray flat pause kurtosis across syntactic boundaries, unnatural digraph timing consistency, and near-zero structural mid-sentence refactoring across multi-hundred-word technical explanations or multi-function source code implementations.

Second-screen visual transcription—where an unproctored contributor displays an external LLM answer on a tablet or adjacent laptop and re-types it manually—preserves organic human finger dwell mechanics while erasing generative composition entropy. Manual transcription edit graphs progress monotonically from character offset zero to end-of-file with exclusively local typographic backspaces (`<= 3` characters) and virtually zero mid-paragraph structural reorganizations, identifier renamings, or multi-clause deletions that inevitably occur when an authentic domain specialist tests intermediate hypotheses, revises symbolic equations, and iteratively refactors function signatures inside a live coding or analytical workspace.

Clipboard Quarantine, Shadow-DOM Event Attestation, and Proctored VDI Sandboxes

Production RLHF workspaces encapsulate text and code editors inside closed Shadow-DOM trees backed by append-only cryptographic event accumulators. Every character delta committed to document state must reconcile against either a browser-attested hardware `KeyboardEvent` carrying `Event.isTrusted === true` or a whitelisted internal IDE transformation such as auto-indentation, bracket completion, or HMAC-signed intra-workspace paste. Whenever serialized submission character counts exceed the authenticated input ledger on the ingestion server, programmatic DOM state mutation by malicious browser extensions, userscripts, or DevTools console hooks is proved deterministically and rejected automatically.

Crude global clipboard blocking frustrates legitimate senior engineers and PhDs who routinely copy function signatures, stack traces, SQL column names, or numerical constants across panes within the annotation task. Normative interfaces enforce cryptographically scoped clipboard quarantine instead: copying any span inside the prompt specification, embedded reference viewer, or sandboxed terminal registers a session-salted SHA-256 digest in memory, allowing frictionless internal reuse while blocking unattested external OS clipboard payloads at the `beforeinput` (`insertFromPaste` and `insertFromDrop`) event interception boundary without penalizing authentic senior developer productivity or editing ergonomics.

Browser focus-lifecycle telemetry (`visibilitychange`, `window.onblur`, and pointer-exit timestamps) exposes multi-window prompt smuggling loops even when external text is manually re-keyed. Contributors who repeatedly select prompt regions, trigger a 15-to-50-second application focus loss, and return to emit uninterrupted 70-WPM prose without scratchpad iteration trigger automated fraud review. For sensitive frontier code, cybersecurity, and proprietary STEM evaluation suites, high-assurance alignment labs eliminate local client-browser ambiguity altogether by streaming hardened annotation IDEs over zero-clipboard, egress-restricted Virtual Desktop Infrastructure (VDI) instances backed by full cryptographic session replay and process isolation.

from dataclasses import dataclass
import math
import statistics
from typing import Sequence

@dataclass(frozen=True)
class TelemetryVerdict:
    passed: bool
    flags: tuple[str, ...]
    iki_cv: float
    revision_ratio: float
    untrusted_char_gap: int

def audit_submission_telemetry(
    final_text: str,
    trusted_iki_ms: Sequence[float],
    total_trusted_insertions: int,
    total_deletions: int,
    structural_cursor_jumps: int,
    blur_events_followed_by_burst: int,
) -> TelemetryVerdict:
    flags: list[str] = []
    emitted_len = len(final_text)
    untrusted_gap = max(0, emitted_len - total_trusted_insertions)

    if untrusted_gap > 16:
        flags.append("dom_or_clipboard_injection_gap")

    if len(trusted_iki_ms) < 40:
        flags.append("insufficient_keystroke_lineage")
        return TelemetryVerdict(False, tuple(flags), 0.0, 0.0, untrusted_gap)

    clipped = [min(x, 8000.0) for x in trusted_iki_ms if x > 0]
    mean_iki = statistics.fmean(clipped)
    stdev_iki = statistics.pstdev(clipped)
    iki_cv = stdev_iki / mean_iki if mean_iki > 0 else 0.0
    long_pauses = sum(1 for x in clipped if x >= 1500.0) / len(clipped)
    revision_ratio = total_deletions / max(1, total_trusted_insertions)

    # Constant-cadence or Gaussian-jittered auto-typers lack natural pause bimodality
    if iki_cv < 0.52 or long_pauses < 0.018:
        flags.append("synthetic_keystroke_cadence")

    # Second-screen visual transcription exhibits near-zero structural editing
    if emitted_len >= 900 and revision_ratio < 0.035 and structural_cursor_jumps < 3:
        flags.append("linear_transcription_signature")

    if blur_events_followed_by_burst >= 2:
        flags.append("external_window_roundtrip_pattern")

    passed = len(flags) == 0 and not math.isnan(iki_cv)
    return TelemetryVerdict(
        passed=passed,
        flags=tuple(flags),
        iki_cv=round(iki_cv, 4),
        revision_ratio=round(revision_ratio, 4),
        untrusted_char_gap=untrusted_gap,
    )

Adversarial Prompt Canaries, Invisible Zero-Width Traps, and Blind Golden Sets

Client-side input telemetry is paired with active cryptographic and semantic prompt canaries engineered to detect external LLM roundtrips regardless of how synthetic text is smuggled back into the workspace. At the presentation layer, clipped off-viewport spans, CSS-minimized `aria-hidden` directives, or zero-width Unicode watermark payloads embedded inside selectable prompt blocks remain invisible to human annotators reading the screen yet copy verbatim into plaintext buffers—instructing external chatbots to embed a deterministic canary function identifier, bespoke six-digit numerical constant, or signature transitional phrase that triggers instant server-side quarantine.

Counterfactual specification traps catch smartphone-camera OCR and visual re-typing workflows where DOM clipboard watermarks never leave the screen. Embedded canary items stipulate synthetic local axioms directly inside the task instructions—such as a bespoke internal RPC library with reversed parameter conventions, a non-standard gravitational boundary condition, or a hypothetical statutory tax bracket—that attentive human domain specialists apply effortlessly from local prompt context while off-the-shelf commercial assistants silently override those novel axioms using dominant high-probability pretraining priors memorized from public open-source codebases, documentation wikis, and standard reference textbooks.

Pairwise DPO preference queues mandate 5% to 10% blind golden-set injection built around adversarial 'eloquent-wrong vs. terse-correct' contrast pairs pre-screened to fool frontier commercial assistants. In every golden trap, Completion A presents a polished, empathetically formatted 550-word synthetic response concealing a fatal race condition, inverted sign, or hallucinated statute, while Completion B delivers an unadorned 140-word human answer verified by unit tests or primary citations. Contributors delegating evaluation to external LLMs systematically misgrade these calibrated traps at empirical failure rates exceeding 80% across blind vendor audits.

Cross-Model Perplexity Ratios, Lexical Idiolect Drift, and Cohort Quarantine

Applying off-the-shelf single-document 'AI prose classifiers' to isolated 200-word technical annotations generates severe false positives against formal mathematical proofs, standardized API docstrings, and non-native English domain experts. High-precision statistical auditing instead evaluates longitudinal distributions across each contributor's submission history, combining cross-tokenizer observer-to-performer log-perplexity ratios (Binoculars-class curvature statistics scored across contemporary open-weight reference models) against verified personal writing, syntax, and keystroke baselines captured during live, camera-proctored technical onboarding work-sample examinations conducted under government-ID and human proctor verification before any production queue access is ever granted.

Authenticated human specialists maintain measurable stylistic invariants across multi-week cohorts: stable syntactic clause nesting depth, characteristic code-comment formatting, preferred variable naming conventions, and consistent discourse-marker frequencies. Whenever a credentialed PhD or staff SWE account is illicitly subcontracted to an unvetted proxy farm or pivoted onto an LLM post-editing workflow mid-milestone, sudden Kullback-Leibler divergence across character n-grams, AST structural idioms, and inter-keystroke cadence distributions reliably flags the compromised contributor account within 10 to 15 completed rows for immediate queue suspension, payout freeze, and live video re-verification.

Normative post-training SOWs enforce whole-lineage contributor quarantine rather than piecemeal row rejection when synthetic fraud is confirmed. If an annotator fails two blind counterfactual prompt canaries or triggers corroborated keystroke-biometric and idiolect-drift alarms, 100% of historical SFT trajectories and DPO pairs attributed to that `annotator_id_hash` across the entire delivery batch are invalidated and re-collected from scratch at vendor expense, ensuring lightly paraphrased synthetic residue never slips past single-row outlier filters to contaminate downstream scalar reward-model calibration, DPO preference margins, or production policy gradient updates.

• Require cryptographic `telemetry_attestation` metadata (IKI CV, revision ratio, untrusted gap, focus-blur count) inside every delivered JSONL record.
• Anchor every contributor's idiolect and typing baseline during a live, camera-proctored domain work-sample exam prior to production queue access.
• Seed >= 7% indistinguishable counterfactual and eloquent-wrong golden canaries across every SFT and DPO production batch.
• Enforce whole-contributor lineage purging (`annotator_id_hash` rollback) whenever synthetic assistance is confirmed on >= 2 canary items.

Summary of Post-Training Engineering Rules

• Undisclosed crowdworker LLM usage induces recursive entropy collapse in SFT and trains DPO reward models to prefer verbose sycophancy over terse accuracy.
• Capture full keystroke inter-arrival intervals, deletion-to-emission ratios, and cursor jump entropy to expose both jittered auto-typers and second-screen transcription.
• Pair closed Shadow-DOM `isTrusted` event ledgers with nonce-scoped internal clipboard allowances so workspace usability stays high while external pastes block.
• Inject counterfactual specification traps and 'eloquent-wrong vs. terse-correct' DPO golden pairs that defeat OCR/camera smuggling as well as raw copy-pasting.
• Quarantine and purge the entire historical batch lineage of any `annotator_id_hash` caught failing synthetic prompt canaries rather than deleting isolated rows.

Related Post-Training Specifications