5 ms·

> We’re currently using Mixtral 8x7B behind our endpoint mistral-small... So 45 billion parameters is what they consider their "small" model? I'm excited to se

by pugio 3y ago

> We’re currently using Mixtral 8x7B behind our endpoint mistral-small... So 45 billion parameters is what they consider their "small" model? I'm excited to see what/if their larger models will be.

miven 3y ago
There seems to be an experimental Mistral Medium model listed among other available model endpoints on [1], the comparison table they give shows that it outmatches 8x7B by a few percent on every benchmark listed [1] https://mistral.ai/news/la-plateforme/ https://mistral.ai/news/la-plateforme/
- mebazaa 3y ago
  It apparently outperforms GPT-4 at WinoGrande as well…
jug 3y ago
Note that it processes tokens at speed and cost of a 12B model though.
- visarga 3y ago
  If it uses 2 experts, they should parallelize so closer to 7B speed?
  - knd775 3y ago
    Memory bandwidth is still a factor, right?
michaelt 3y ago
> So 45 billion parameters is what they consider their "small" model? According to Wikipedia: Rumors claim that GPT-4 has 1.76 trillion parameters, which was first estimated by the speed it was running and by George Hotz. [1] [1] https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ https://the-decoder.com/gpt-4-architecture-datasets-costs-an...
- GaggiX 3y ago
  I still need to understand how George Hotz knows about the GPT-4 architecture if it is true.
  - deleted 3y ago
    [deleted]
  - PedroBatista 3y ago
    You don't necessarily need to know the architecture, given the "only" real metric regarding speed is tokens/sec and that pretty much depends on memory bandwidth, you can infer with some certainty the size of the model. Also, if we have been eating up posted "benchmarks" with no way to independently validate them and watching heavily edited video presentations, why can't we trust our wonder kid?
    - GaggiX 3y ago
      That doesn't explain how we know that GPT-4 is a sparse MoE model with X experts of Y size and using Z of them during inference.
      - Laaas 3y ago
        IIRC it was leaked/confirmed by accident or something like that
  - sigmoid10 3y ago
    As someone who has worked in the field for many years now and closely follows not just the engineering side but also the academic literature and the personell movements on linkedin, I too was able to put together a lot of this. Especially with GPT-3 Turbo it was obvious what they did due to the speed difference. At least in terms of model architectures and orders of magnitude for parameters. From there you could do some back of the envelope calculations and guess how big GPT4 had to be given its speed. I wouldn't have dared to say any specific numbers with authority, but maybe Hotz has talked to someone at OpenAI. On the other hand, the updated article now claims his numbers were off by a factor of 2 (at least for the individual experts - he still got the total number of parameters right). So yeah, maybe he was just guessing like the rest of us after all.
deleted 3y ago
[deleted]
pbmonster 3y ago
It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers. Which also means you can fit the 8 models in a much smaller amount of memory than a 45B model. Latency will also be much smaller than a 45B model, since the next token is always only created by 2 of the 8 models (which 2 models are run is chosen by a different, even smaller/faster, model).
- airgapstopgap 3y ago
  > It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers. No, Mixture-of-Experts is not stacking finetunes of the same base model.
  - pbmonster 3y ago
    Do you have any more information on the topic? I remember reading that about significant memory savings achieved by reusing most layers. Made sense to mee on first sight to me, because you don't need to train stuff like syntax and grammar 8 times in 8 different ways. Also would explain why interference of two 7B models has the cost of running a 12B model.
    - airgapstopgap 3y ago
      The original paper by Shazeer suffices. What you are saying is in theory possible to do and may have been done in practice here, but in the general case MoE is trained from scratch and specializations of layers which develop are not products of some design choice.
patapong 3y ago
They have a description and performance evaluation of Mistral-medium on their website: https://mistral.ai/news/la-plateforme/ https://mistral.ai/news/la-plateforme/ "Our highest-quality endpoint currently serves a prototype model, that is currently among the top serviced models available based on standard benchmarks. It masters English/French/Italian/German/Spanish and code and obtains a score of 8.6 on MT-Bench."

by pugio 3y ago

> We’re currently using Mixtral 8x7B behind our endpoint mistral-small... So 45 billion parameters is what they consider their "small" model? I'm excited to see what/if their larger models will be.

miven 3y ago

There seems to be an experimental Mistral Medium model listed among other available model endpoints on [1], the comparison table they give shows that it outmatches 8x7B by a few percent on every benchmark listed [1] https://mistral.ai/news/la-plateforme/ https://mistral.ai/news/la-plateforme/

mebazaa 3y ago
It apparently outperforms GPT-4 at WinoGrande as well…

jug 3y ago

Note that it processes tokens at speed and cost of a 12B model though.

visarga 3y ago
If it uses 2 experts, they should parallelize so closer to 7B speed?
- knd775 3y ago
  Memory bandwidth is still a factor, right?

michaelt 3y ago

> So 45 billion parameters is what they consider their "small" model? According to Wikipedia: Rumors claim that GPT-4 has 1.76 trillion parameters, which was first estimated by the speed it was running and by George Hotz. [1] [1] https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ https://the-decoder.com/gpt-4-architecture-datasets-costs-an...

GaggiX 3y ago
I still need to understand how George Hotz knows about the GPT-4 architecture if it is true.
- deleted 3y ago
  [deleted]
- PedroBatista 3y ago
  You don't necessarily need to know the architecture, given the "only" real metric regarding speed is tokens/sec and that pretty much depends on memory bandwidth, you can infer with some certainty the size of the model. Also, if we have been eating up posted "benchmarks" with no way to independently validate them and watching heavily edited video presentations, why can't we trust our wonder kid?
  - GaggiX 3y ago
    That doesn't explain how we know that GPT-4 is a sparse MoE model with X experts of Y size and using Z of them during inference.
    - Laaas 3y ago
      IIRC it was leaked/confirmed by accident or something like that
- sigmoid10 3y ago
  As someone who has worked in the field for many years now and closely follows not just the engineering side but also the academic literature and the personell movements on linkedin, I too was able to put together a lot of this. Especially with GPT-3 Turbo it was obvious what they did due to the speed difference. At least in terms of model architectures and orders of magnitude for parameters. From there you could do some back of the envelope calculations and guess how big GPT4 had to be given its speed. I wouldn't have dared to say any specific numbers with authority, but maybe Hotz has talked to someone at OpenAI. On the other hand, the updated article now claims his numbers were off by a factor of 2 (at least for the individual experts - he still got the total number of parameters right). So yeah, maybe he was just guessing like the rest of us after all.

deleted 3y ago

[deleted]

pbmonster 3y ago

It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers. Which also means you can fit the 8 models in a much smaller amount of memory than a 45B model. Latency will also be much smaller than a 45B model, since the next token is always only created by 2 of the 8 models (which 2 models are run is chosen by a different, even smaller/faster, model).

airgapstopgap 3y ago
> It's not even close to a 45B model. They trained 8 different fine-tunes on the same base model. This means the 8 models differ only by a couple of layers and share the rest of their layers. No, Mixture-of-Experts is not stacking finetunes of the same base model.
- pbmonster 3y ago
  Do you have any more information on the topic? I remember reading that about significant memory savings achieved by reusing most layers. Made sense to mee on first sight to me, because you don't need to train stuff like syntax and grammar 8 times in 8 different ways. Also would explain why interference of two 7B models has the cost of running a 12B model.
  - airgapstopgap 3y ago
    The original paper by Shazeer suffices. What you are saying is in theory possible to do and may have been done in practice here, but in the general case MoE is trained from scratch and specializations of layers which develop are not products of some design choice.

patapong 3y ago

They have a description and performance evaluation of Mistral-medium on their website: https://mistral.ai/news/la-plateforme/ https://mistral.ai/news/la-plateforme/ "Our highest-quality endpoint currently serves a prototype model, that is currently among the top serviced models available based on standard benchmarks. It masters English/French/Italian/German/Spanish and code and obtains a score of 8.6 on MT-Bench."