Primary Post-Training, Reward Modeling & RLVR Literature
Every optimization objective, step-supervision protocol, reward-evaluation benchmark, and containerized sandbox specification on bestrlhfdata.com derives from peer-reviewed alignment research and canonical open-source trainer implementations.
Foundational Papers, Benchmarks & Frameworks
Training language models to follow instructions with human feedback (InstructGPT)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Paul Christiano, Jan Leike, Ryan Lowe (NeurIPS). Establishes the canonical three-stage post-training architecture—human SFT demonstrations, Bradley-Terry scalar reward modeling over ranked completions, and KL-penalized PPO policy updates—proving alignment signal dominates raw parameter count.
[PRIMARY_PAPER ↗]Constitutional AI: Harmlessness from AI Feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Dario Amodei, Jared Kaplan et al. (Anthropic / arXiv). Formalizes explicit principle-based critique-and-revision loops and helpful-versus-harmless Pareto evaluation, informing non-evasive refusal tone rubrics and explicit constitution axes.
[PRIMARY_PAPER ↗]Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (NeurIPS). Derives the closed-form mapping between KL-constrained reward maximization and optimal policy log-probability ratios, eliminating separate critic instability during pairwise preference optimization.
[PRIMARY_PAPER ↗]Let's Verify Step by Step (PRM800K)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe (OpenAI / ICLR). Demonstrates on competition MATH that supervising discrete intermediate reasoning steps (`+1 / 0 / -1`) decisively outperforms outcome-only reward models and mitigates spurious positive derivations.
[PRIMARY_PAPER ↗]SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan (ICLR). Defines the standard containerized repository evaluation and RLVR harness pairing real GitHub pull requests with deterministic `FAIL_TO_PASS` and `PASS_TO_PASS` unit-test execution.
[PRIMARY_PAPER ↗]RewardBench: Evaluating Reward Models for Language Modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi (AllenAI / NAACL Findings). Provides controlled contrast sets across chat, reasoning, code, and adversarial 'chat-hard' traps exposing length bias, sycophancy, and subtle factual blind spots in neural reward models.
[PRIMARY_PAPER ↗]DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, Daya Guo (arXiv). Introduces Group Relative Policy Optimization (GRPO), estimating intra-group rollout advantages without memory-heavy value critics and powering large-scale RLVR across math and code.
[PRIMARY_PAPER ↗]TRL — Transformer Reinforcement Learning Documentation (SFT, DPO, KTO, PRM & GRPO)
Leandro von Werra, Quentin Gallouedec, Lewis Tunstall & Hugging Face Alignment Team. Canonical PyTorch implementation reference defining standard dataset schemas, assistant token loss masking, and distributed rollout trainers for SFT, DPO, SimPO, KTO, PRM, PPO, and GRPO.
[PRIMARY_PAPER ↗]Evidentiary Bar for Alignment Objectives and Benchmark Harnesses
Best RLHF Data restricts normative citations to peer-reviewed machine learning venues (NeurIPS, ICLR, ICML, NAACL) and open-source post-training repositories providing reproducible loss equations, container build manifests, and public ablation curves.
Unreplicated marketing blog claims and closed vendor whitepapers lacking public mathematical specifications or verifiable test harnesses are excluded from our methodology standards.
Translating Alignment Theory into Production JSONL Contracts
Theoretical papers derive loss functions assuming clean preference triples `(x, y_w, y_l)` or noise-free step rewards. Our normative downloadable schemas (`/artifacts/dpo-preference-pair.jsonl` and `/artifacts/rlhf-rubric-template.json`) operationalize Rafailov et al., Lightman et al., Jimenez et al., and Shao et al. by binding reward labels directly to container exit codes, length-penalty formulas, and anti-LLM telemetry attestations.