3 ms·

Would it be correct to summarize that the general conceptual shift is optimizing MOEs on more specific smaller tasks? It smells like borderline overfitting to m

• whimsicalism 2y ago
  the core insight is that you can train on extremely sparse deterministic reward signal and it just works that MoEs are better for a given compute budget has been known for a while.
  • nick3443 2y ago
    Thank you for the reply. Maybe I'm daft or out of the loop, I don't see the difference between this method and what I have seen previously as "fine tuning on synthetic data". They also mention in the source tweet that the quality of the base model is important, which is not really accounted for in the $30 figure.
    • whimsicalism 2y ago
      'finetuning on synthetic data' = same LM task with BCE objective, ie. training on next word prediction on good text. 'rl on sparse rewards' = RL objective - reward is given by deterministic evaluation, reward is passed back as sparse signal. there is no 'ground truth' data to compare against.
      • nick3443 2y ago
        Ah now I see the difference. Thank you!!