5 ms·

This paper details the model that's been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It's roughly sized at 13B, and ranked much,

This paper details the model that's been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It's roughly sized at 13B, and ranked much, much higher than competitively sized models by, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1916896/llm_comparisontest_confirm_leaderboard_big_news/ https://www.reddit.com/r/LocalLLaMA/comments/1916896/llm_com.... Ravenwolf notes that the model slightly outperforms some of its benchmark testing, and this is my experience. It's surprisingly good for a model of its size, and a very capable daily driver on a Mac for chat, code input and other uses. Something that has come to light since the release of the weights, and not mentioned in this paper is that it looks like fairly likely that the 8 experts were all seeded by Mistral 7B and subsequently diverged. This has generated a lot of experimentation in the local LLM community with cloning models as a way to cheaply generate experts. It was generally thought likely that training an 8x7B network would be as much work as training 8 7B networks, but this seems not to have been true for Mistral, which is super interesting. There's still a lot of rapid innovation happening in this space, with papers like Calm from DeepMind this week, and a lot of the adhoc experimental layer combining happening in the wild, (see, e.g. Goliath-120b), I think we're likely to see some pretty interesting architectural improvements this year in the LLM space. Calm seems to point the way to a next step after MoE, and models like Goliath seem to indicate that even a really really lazy version of Calm (no Linear layer combination, just literally alternating layers at full weights) can be very impactful. Overall I think we will see really, really strong models that are performant on consumer hardware in 2024, likely first half of this year.

Casteil 3y ago
I've had excellent results with Mixtral too - it's genuinely impressive. Only problem is that it's a relatively big model that's difficult to run with full GPU inference on consumer hardware (vs the 7b/13b models people typically use). So far, the main consumer platform capable of running it without 'ruining' the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally >=48GB. It can apparently be done on 32 or 36GB, but there's not much headroom. Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.
- ignoramous 3y ago
  > the newer Apple Silicon Macs with unified memory Mixtral has been MLXd already? Write ups, if any?
  - LoganDark 3y ago
    Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
    - mkesper 3y ago
      But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
      - zarzavat 3y ago
        Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
        LoganDark 3y ago
        Not to mention that's 128GB for a single GPU. No need to shard or split.
  - vessenes 3y ago
    Yes it has, actually: https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples. It's right in the main repo. NB, I haven't tried this, I'm using llama.cpp with a non-K-quant quantization on my MBP.
    - 3y ago
bugglebeetle 3y ago
Mixtral is good but those Ravenwolf benchmarks are meaningless. It’s like some random dude trying to reinvent MMLU without any rigor or consistency and in German. Dataset contamination is a problem, but not one that’s solved by folkloric evaluation of LLMs by people asking for tips on a subreddit.
- vessenes 3y ago
  I don't think they're meaningless; they have a few benefits: 1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed 2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests 3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning 4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena 5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases. Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations. That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.
  - bugglebeetle 3y ago
    Any tests that are unfalsifiable and can’t be reproduced are meaningless when it comes to gauging the performance of LLMs (and most other things). I could also post on a subreddit and say I have a secret set of German tests that may or may not exist and and I like these models, but that does nothing to advance the science of evaluating these things. If you want to evaluate human preferences, you can use chatbot arena, which can be gamed, but at least reflects more than what one guy says to be true. And this is with me agreeing that Nous-Capybara is also a good model. But don’t take my word for it because it’s not worth very much!
    - vessenes 3y ago
      I think we agree on almost everything about this - which is to say, probably both you and I think the Wolfram Ravenwolf tests are just about useful enough to indicate whether or not a model might be worth downloading, but certainly not enough to justify spending money, say, or planning around. So, yes, I'm with you. I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans. However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.
tracerbulletx 3y ago
I'm looking forward to all the hardware announcements. It's certainly looking like intentionally designed on device acceleration of LLMs for consumers is coming.
pseudosavant 3y ago
I'm really curious if/when we'll see MoE based on even smaller models like Phi-2?
- lucidyan 3y ago
  Already https://nitter.net/maximelabonne/status/1744867841436700850 https://nitter.net/maximelabonne/status/1744867841436700850
- Redster 3y ago
  At this small a scale, what would the benefits be of something like a 8x2B as opposed to moving to a 7B?
  - pseudosavant 3y ago
    For the output performance (quality and tokens/second), Mixtral 8x7B seems to require less VRAM. But it is still hard to make it fit, even with a lot of quantization, within the GPU RAM of most discreet consumer GPUs. Perhaps a smaller base model like Phi-2 could bring the VRAM requirements down, but the MoE will bring the output quality up from Phi-2.

Casteil 3y ago

I've had excellent results with Mixtral too - it's genuinely impressive. Only problem is that it's a relatively big model that's difficult to run with full GPU inference on consumer hardware (vs the 7b/13b models people typically use). So far, the main consumer platform capable of running it without 'ruining' the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally >=48GB. It can apparently be done on 32 or 36GB, but there's not much headroom. Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.

ignoramous 3y ago
> the newer Apple Silicon Macs with unified memory Mixtral has been MLXd already? Write ups, if any?
- LoganDark 3y ago
  Not to my knowledge. But because the unified memory doubles as VRAM for the onboard GPU, normal GPU acceleration can access the entire model even if it's 50+ GB. That's why ASi Macs are currently the holy grail for at-home inferencing, and also why projects like llama.cpp focus so much on ASi above all else, and why so many UIs release for macOS first before other operating systems. Certain Mac models offer up to 192GB of unified memory.
  - mkesper 3y ago
    But that's not a Macbook. And a Macbook M3Max with 128GB of RAM is almost 8000€.
    - zarzavat 3y ago
      Considering how inaccessible and expensive 128GB of pro-level cards is, that is believe it or not, a good price.
      - LoganDark 3y ago
        Not to mention that's 128GB for a single GPU. No need to shard or split.
- vessenes 3y ago
  Yes it has, actually: https://github.com/ml-explore/mlx-examples https://github.com/ml-explore/mlx-examples. It's right in the main repo. NB, I haven't tried this, I'm using llama.cpp with a non-K-quant quantization on my MBP.
  - 3y ago

bugglebeetle 3y ago

Mixtral is good but those Ravenwolf benchmarks are meaningless. It’s like some random dude trying to reinvent MMLU without any rigor or consistency and in German. Dataset contamination is a problem, but not one that’s solved by folkloric evaluation of LLMs by people asking for tips on a subreddit.

vessenes 3y ago
I don't think they're meaningless; they have a few benefits: 1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed 2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests 3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning 4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena 5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases. Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations. That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.
- bugglebeetle 3y ago
  Any tests that are unfalsifiable and can’t be reproduced are meaningless when it comes to gauging the performance of LLMs (and most other things). I could also post on a subreddit and say I have a secret set of German tests that may or may not exist and and I like these models, but that does nothing to advance the science of evaluating these things. If you want to evaluate human preferences, you can use chatbot arena, which can be gamed, but at least reflects more than what one guy says to be true. And this is with me agreeing that Nous-Capybara is also a good model. But don’t take my word for it because it’s not worth very much!
  - vessenes 3y ago
    I think we agree on almost everything about this - which is to say, probably both you and I think the Wolfram Ravenwolf tests are just about useful enough to indicate whether or not a model might be worth downloading, but certainly not enough to justify spending money, say, or planning around. So, yes, I'm with you. I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans. However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.

tracerbulletx 3y ago

I'm looking forward to all the hardware announcements. It's certainly looking like intentionally designed on device acceleration of LLMs for consumers is coming.

pseudosavant 3y ago

I'm really curious if/when we'll see MoE based on even smaller models like Phi-2?

lucidyan 3y ago
Already https://nitter.net/maximelabonne/status/1744867841436700850 https://nitter.net/maximelabonne/status/1744867841436700850
Redster 3y ago
At this small a scale, what would the benefits be of something like a 8x2B as opposed to moving to a 7B?
- pseudosavant 3y ago
  For the output performance (quality and tokens/second), Mixtral 8x7B seems to require less VRAM. But it is still hard to make it fit, even with a lot of quantization, within the GPU RAM of most discreet consumer GPUs. Perhaps a smaller base model like Phi-2 could bring the VRAM requirements down, but the MoE will bring the output quality up from Phi-2.