Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, Sharma, Mitchell, Ermon, Manning, Finn
2023 NeurIPS 2023
TL;DR
Derive a closed-form loss that optimizes a policy against preference data without training a separate reward model or running PPO. Much simpler than RLHF, competitive quality.