2 ms·

I can second that. From what I’ve heard from people at leading labs, it’s not clear that dpo is worth switching to from RLHF