4 ms·

This entire difference can be explained due to their double mixture of experts architecture. So only 1/4 MLP and attention blocks are used at any time. Maybe th

This entire difference can be explained due to their double mixture of experts architecture. So only 1/4 MLP and attention blocks are used at any time. Maybe this should be the headline, Moe reduces compute by a factor of 4 without losing accuracy. But this is already known. Still interesting to see a smaller Moe model. This could be the ideal size for many local applications.

brianarbuckle 3y ago
Agreed. In inference, I could even imagine a 4-bit conversion for certain edge devices.
Centigonal 3y ago
MoE reduces compute cost for inference at scale, but not for training. You still have to train the whole model (plus the router)
- rileyphone 3y ago
  With an MoE you only need to train a smaller model which you can then combine into an x8 and finetune/train the router. Mistral used their 7B base to make Mixtral, Qwen's new MoE uses their 1.8B model upscaled to 2.7B, pretty sure Grok also trained a smaller model first.
  - samus 3y ago
    You're only correct about Qwen's MoE. I presume that Chinese model builders feel more pressure to be efficient about using their GPU time because of sanctions.
  - phree_radical 3y ago
    Very incorrect! The "8x7b" in the name regularly confuses people into some similar conclusion, but there are not eight 7b "experts" in Mixtral 8x. It's more apt to think of all 256 FFN's as the "experts," as each expert FFN on a given layer has no relation to the expert FFN's on other layers. You need to train them all within the MoE architecture, as combining existing models ("clown car MoE") works, but isn't gaining anything from the architecture/sparsity
    - epups 3y ago
      Sorry, could you expand on this a bit further? Are you saying that for a MoE, you want to train the exact same model, and then just finetune the feed forward networks differently for each of them? And you're saying that separately training 8 different models would not be efficient - do we have evidence for that?
- Me1000 3y ago
  It’s absolutely beneficial when training because the forward pass and back propagation is still only on the neurons that were activated. The Mistral guys specifically mention that training speed (due to not needing as much compute) was one of the reasons Mixtral was released so soon after Mistal 7b.
smusamashah 3y ago
If MoEs are that good, we know GPT-4 is, than why not train very specific MoEs. One part of MoE could be a perfect Math model which can actually calculate 2+2. Wouldn't models like these be better in general?
- refulgentis 3y ago
  Keeping it short: "Not even wrong", in the Pauli sense. - People hear "mixture of experts" and they think "N specialists" - but ex. think how much know you need to know to autocomplete "Two plus two is " - Fundamental thing of ML is you define functions and give it data, and the more data you give it to the better. Once youre at "I will simply give it the training data needed to be good enough at the task and wall off that part of the implementation" you're outside ML and have a chicken and egg problem - We don't know GPT-4 is MoE - MoE in practice is fundamentally about trading off runtime vs. static size properties to gain inference speed. I.e. 7x8 stored and picking 7x2 at runtime means youre somewhere between 7x2 and 7x3 in quality, inference at 7x2 speed, and have to train and store and load 7x8. You don't reach for it to increase quality, you reach for it to increase inference speed at the expense of inference ram and total model size.
  - viraptor 3y ago
    > We don't know GPT-4 is MoE Didn't Yampleg's tweet / leak confirm this one? I mean, he could be wrong about this, but I thought the consensus was on it being true by now. (Copy of the removed tweets at https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_details_are_leaked/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_de... )
    - refulgentis 3y ago
      It's just a dude retweeting a substack. I wouldn't bet against it* but I wouldn't bet on it either. His tweet would have just linked to the article in the top comment. * I used to crusade against this rumor because the only source is that article, and people repeating that source. But I imagine it's a no-brainer given they have enough users that they essentially get a throughput bump 'for free' even if the model weights are huge, i.e. better to utilize as much GPU ram as you can muster, the cost of needing more GPU ram is offset by the cost of being able to run multiple inference against the model all the time anyway

brianarbuckle 3y ago

Agreed. In inference, I could even imagine a 4-bit conversion for certain edge devices.

Centigonal 3y ago

MoE reduces compute cost for inference at scale, but not for training. You still have to train the whole model (plus the router)

rileyphone 3y ago
With an MoE you only need to train a smaller model which you can then combine into an x8 and finetune/train the router. Mistral used their 7B base to make Mixtral, Qwen's new MoE uses their 1.8B model upscaled to 2.7B, pretty sure Grok also trained a smaller model first.
- samus 3y ago
  You're only correct about Qwen's MoE. I presume that Chinese model builders feel more pressure to be efficient about using their GPU time because of sanctions.
- phree_radical 3y ago
  Very incorrect! The "8x7b" in the name regularly confuses people into some similar conclusion, but there are not eight 7b "experts" in Mixtral 8x. It's more apt to think of all 256 FFN's as the "experts," as each expert FFN on a given layer has no relation to the expert FFN's on other layers. You need to train them all within the MoE architecture, as combining existing models ("clown car MoE") works, but isn't gaining anything from the architecture/sparsity
  - epups 3y ago
    Sorry, could you expand on this a bit further? Are you saying that for a MoE, you want to train the exact same model, and then just finetune the feed forward networks differently for each of them? And you're saying that separately training 8 different models would not be efficient - do we have evidence for that?
Me1000 3y ago
It’s absolutely beneficial when training because the forward pass and back propagation is still only on the neurons that were activated. The Mistral guys specifically mention that training speed (due to not needing as much compute) was one of the reasons Mixtral was released so soon after Mistal 7b.

smusamashah 3y ago

If MoEs are that good, we know GPT-4 is, than why not train very specific MoEs. One part of MoE could be a perfect Math model which can actually calculate 2+2. Wouldn't models like these be better in general?

refulgentis 3y ago
Keeping it short: "Not even wrong", in the Pauli sense. - People hear "mixture of experts" and they think "N specialists" - but ex. think how much know you need to know to autocomplete "Two plus two is " - Fundamental thing of ML is you define functions and give it data, and the more data you give it to the better. Once youre at "I will simply give it the training data needed to be good enough at the task and wall off that part of the implementation" you're outside ML and have a chicken and egg problem - We don't know GPT-4 is MoE - MoE in practice is fundamentally about trading off runtime vs. static size properties to gain inference speed. I.e. 7x8 stored and picking 7x2 at runtime means youre somewhere between 7x2 and 7x3 in quality, inference at 7x2 speed, and have to train and store and load 7x8. You don't reach for it to increase quality, you reach for it to increase inference speed at the expense of inference ram and total model size.
- viraptor 3y ago
  > We don't know GPT-4 is MoE Didn't Yampleg's tweet / leak confirm this one? I mean, he could be wrong about this, but I thought the consensus was on it being true by now. (Copy of the removed tweets at https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_details_are_leaked/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/gpt4s_de... )
  - refulgentis 3y ago
    It's just a dude retweeting a substack. I wouldn't bet against it* but I wouldn't bet on it either. His tweet would have just linked to the article in the top comment. * I used to crusade against this rumor because the only source is that article, and people repeating that source. But I imagine it's a no-brainer given they have enough users that they essentially get a throughput bump 'for free' even if the model weights are huge, i.e. better to utilize as much GPU ram as you can muster, the cost of needing more GPU ram is offset by the cost of being able to run multiple inference against the model all the time anyway