3 ms·

If MoEs are that good, we know GPT-4 is, than why not train very specific MoEs. One part of MoE could be a perfect Math model which can actually calculate 2+2.

If MoEs are that good, we know GPT-4 is, than why not train very specific MoEs. One part of MoE could be a perfect Math model which can actually calculate 2+2. Wouldn't models like these be better in general?

refulgentis 3y ago
Keeping it short: "Not even wrong", in the Pauli sense. - People hear "mixture of experts" and they think "N specialists" - but ex. think how much know you need to know to autocomplete "Two plus two is " - Fundamental thing of ML is you define functions and give it data, and the more data you give it to the better. Once youre at "I will simply give it the training data needed to be good enough at the task and wall off that part of the implementation" you're outside ML and have a chicken and egg problem - We don't know GPT-4 is MoE - MoE in practice is fundamentally about trading off runtime vs. static size properties to gain inference speed. I.e. 7x8 stored and picking 7x2 at runtime means youre somewhere between 7x2 and 7x3 in quality, inference at 7x2 speed, and have to train and store and load 7x8. You don't reach for it to increase quality, you reach for it to increase inference speed at the expense of inference ram and total model size.
- viraptor 3y ago
  > We don't know GPT-4 is MoE Didn't Yampleg's tweet / leak confirm this one? I mean, he could be wrong about this, but I thought the consensus was on it being true by now. (Copy of the removed tweets at https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_details_are_leaked/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_de... )
  - refulgentis 3y ago
    It's just a dude retweeting a substack. I wouldn't bet against it* but I wouldn't bet on it either. His tweet would have just linked to the article in the top comment. * I used to crusade against this rumor because the only source is that article, and people repeating that source. But I imagine it's a no-brainer given they have enough users that they essentially get a throughput bump 'for free' even if the model weights are huge, i.e. better to utilize as much GPU ram as you can muster, the cost of needing more GPU ram is offset by the cost of being able to run multiple inference against the model all the time anyway

refulgentis 3y ago

Keeping it short: "Not even wrong", in the Pauli sense. - People hear "mixture of experts" and they think "N specialists" - but ex. think how much know you need to know to autocomplete "Two plus two is " - Fundamental thing of ML is you define functions and give it data, and the more data you give it to the better. Once youre at "I will simply give it the training data needed to be good enough at the task and wall off that part of the implementation" you're outside ML and have a chicken and egg problem - We don't know GPT-4 is MoE - MoE in practice is fundamentally about trading off runtime vs. static size properties to gain inference speed. I.e. 7x8 stored and picking 7x2 at runtime means youre somewhere between 7x2 and 7x3 in quality, inference at 7x2 speed, and have to train and store and load 7x8. You don't reach for it to increase quality, you reach for it to increase inference speed at the expense of inference ram and total model size.

viraptor 3y ago
> We don't know GPT-4 is MoE Didn't Yampleg's tweet / leak confirm this one? I mean, he could be wrong about this, but I thought the consensus was on it being true by now. (Copy of the removed tweets at https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_details_are_leaked/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_de... )
- refulgentis 3y ago
  It's just a dude retweeting a substack. I wouldn't bet against it* but I wouldn't bet on it either. His tweet would have just linked to the article in the top comment. * I used to crusade against this rumor because the only source is that article, and people repeating that source. But I imagine it's a no-brainer given they have enough users that they essentially get a throughput bump 'for free' even if the model weights are huge, i.e. better to utilize as much GPU ram as you can muster, the cost of needing more GPU ram is offset by the cost of being able to run multiple inference against the model all the time anyway