2 ms·

Iterative reasoning preference optimization

• Jimmc414 3y ago
  Create chain-of-thought candidates with LLM, form preference pairs depending on whether the answers are correct, train using DPO and NLL, then repeat the cycle. Apparently this takes Llama-2-70B-chat from 55.6% to 81.6% on the GSM8k benchmark
  • careai 3y ago
    thank you for your service
  • popinman322 3y ago
    Also, similar to Orca-Math but without a teacher model. They also followed an iterative DPO/KTO scheme, but with no length normalized NLL loss term.
    • algo_trader 3y ago
      If we had a magical (fast) oracle for grading responses, have people done search/expert iteration for LLMs? Specifically for codegen, i am playing with an iterative interpreter that can quickly (re)evaluate a tree of similar responses
• deleted 3y ago
  [deleted]