12 ms·
Mistral 7B is very cool for its size. But unfortunately no open model is close to GPT4 as of right now.
by redox99 3y ago
Mistral 7B is very cool for its size. But unfortunately no open model is close to GPT4 as of right now.
- tarruda 3y agoIf the rumors around GPT4 being a mixture of expert models are true, the this comparison is not fair. What would be interesting is compare GPT4 at a certain task with a small model fine tuned for that task.
- johnthewise 3y agoGPT4 being a mixture of experts is irrelevant imo like we don't care about how many layers there are in a network and how wide those layers are or which type of activation functions are actually used etc. all that matters are we can run it on a specific hardware and the results.
- redox99 3y agoExactly. I don't get why people (non AI researchers) discount MoE like they are cheating or fake parameters. Even if each inference pass only runs part of the network, there's still a trillion learnable parameters there lol.