4 ms·

I thought it was revealed to be fundamentally ensemblamatic in a way the others weren’t? Using “experts” I think? Seems like it would meet the bar for “secret s

by bbor 3y ago

I thought it was revealed to be fundamentally ensemblamatic in a way the others weren’t? Using “experts” I think? Seems like it would meet the bar for “secret sauce” to me

famouswaffles 3y ago
Sparse MoE models are neither new nor secret. The only reason you haven't seen much use of them for LLMs is because they would typically well underperform their dense counterparts. Until this paper (https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2305.14705) indicated they apparently benefit far more from Instruct tuning than dense models, it was mostly a "good on paper" kind of thing. In the paper, you can see the underperformance i'm talking about. Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after. Flan 62b scores 55% before Instruct tuning and 59% after.
- cubefox 3y ago
  This paper came out well after GPT-4, so apparently this was indeed a secret before then.
  - famouswaffles 3y ago
    The user I was replying to was talking about the now and future. We also have no indication sparse models outperform dense counterparts so it's scale either way.
  - HeWhoLurksLate 3y ago
    Is there a difference here between a secret and an unknown? It may well be that some researcher / comp engineer had an idea, tried it out, realized it was incredibly powerful, implemented it for real this time and then published findings after they were sure of it? I'm more of a mechanical engineering adjacent professional than a programmer and only follow AI developments loosely
  - l33tman 3y ago
    The quoted paper yes, but the MoE concept and layers and training is old. Published as a conference paper at ICLR 2017 OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean

by bbor 3y ago

I thought it was revealed to be fundamentally ensemblamatic in a way the others weren’t? Using “experts” I think? Seems like it would meet the bar for “secret sauce” to me

famouswaffles 3y ago

Sparse MoE models are neither new nor secret. The only reason you haven't seen much use of them for LLMs is because they would typically well underperform their dense counterparts. Until this paper (https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2305.14705) indicated they apparently benefit far more from Instruct tuning than dense models, it was mostly a "good on paper" kind of thing. In the paper, you can see the underperformance i'm talking about. Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after. Flan 62b scores 55% before Instruct tuning and 59% after.

cubefox 3y ago
This paper came out well after GPT-4, so apparently this was indeed a secret before then.
- famouswaffles 3y ago
  The user I was replying to was talking about the now and future. We also have no indication sparse models outperform dense counterparts so it's scale either way.
- HeWhoLurksLate 3y ago
  Is there a difference here between a secret and an unknown? It may well be that some researcher / comp engineer had an idea, tried it out, realized it was incredibly powerful, implemented it for real this time and then published findings after they were sure of it? I'm more of a mechanical engineering adjacent professional than a programmer and only follow AI developments loosely
- l33tman 3y ago
  The quoted paper yes, but the MoE concept and layers and training is old. Published as a conference paper at ICLR 2017 OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean