3 ms·
This paper came out well after GPT-4, so apparently this was indeed a secret before then.
by cubefox 3y ago
This paper came out well after GPT-4, so apparently this was indeed a secret before then.
- famouswaffles 3y agoThe user I was replying to was talking about the now and future. We also have no indication sparse models outperform dense counterparts so it's scale either way.
- HeWhoLurksLate 3y agoIs there a difference here between a secret and an unknown? It may well be that some researcher / comp engineer had an idea, tried it out, realized it was incredibly powerful, implemented it for real this time and then published findings after they were sure of it? I'm more of a mechanical engineering adjacent professional than a programmer and only follow AI developments loosely
- l33tman 3y agoThe quoted paper yes, but the MoE concept and layers and training is old. Published as a conference paper at ICLR 2017 OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean