3 ms·

Mistral's latest just released model is well below GPT-3 out of the box. I've seen people speculate that with fine-tuning and RLHF you could get GPT-3 like per

Mistral's latest just released model is well below GPT-3 out of the box. I've seen people speculate that with fine-tuning and RLHF you could get GPT-3 like performance out of it but it's still too early to tell. I'm in agreement with you, I've been following this field for a decade now and GPT-4 did seem to cross a magical threshold for me where it was finally good enough to not just be a curiosity but a real tool. I try to test every new model I can get my hands on and it remains the only one to cross that admittedly subjective threshold for me.

rmbyrro 3y ago
Still, for a 7B model, this is quite impressive.
espadrine 3y ago
> Mistral's latest just released model is well below GPT-3 out of the box The early information I see implies it is above. Mind you, that is mostly because GPT-3 was comparatively low: for instance its 5-shot MMLU score was 43.9%, while Llama2 70B 5-shot was 68.9%[0]. Early benchmarks[1] give Mixtral scores above Llama2 70B on MMLU (and other benchmarks), thus transitively, it seems likely to be above GPT-3. Of course, GPT-3.5 has a 5-shot score of 70, and it is unclear yet whether Mixtral is above or below, and clearly it is below GPT-4’s 86.5. The dust needs to settle, and the official inference code needs to be released, before there is certainty on its exact strength. (It is also a base model, not a chat finetune; I see a lot of people saying it is worse, simply because they interact with it as if it was a chatbot.) [0]: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu https://paperswithcode.com/sota/multi-task-language-understa... [1]: https://github.com/open-compass/MixtralKit#comparison-with-other-models https://github.com/open-compass/MixtralKit#comparison-with-o...
brucethemoose2 3y ago
Have you played with finetunes, like Cybertron? Augmented in wrappers and retrievers like GPT is? It's not there yet, but its waaaay closer than the plain Mistral chat release.

rmbyrro 3y ago

Still, for a 7B model, this is quite impressive.

espadrine 3y ago

> Mistral's latest just released model is well below GPT-3 out of the box The early information I see implies it is above. Mind you, that is mostly because GPT-3 was comparatively low: for instance its 5-shot MMLU score was 43.9%, while Llama2 70B 5-shot was 68.9%[0]. Early benchmarks[1] give Mixtral scores above Llama2 70B on MMLU (and other benchmarks), thus transitively, it seems likely to be above GPT-3. Of course, GPT-3.5 has a 5-shot score of 70, and it is unclear yet whether Mixtral is above or below, and clearly it is below GPT-4’s 86.5. The dust needs to settle, and the official inference code needs to be released, before there is certainty on its exact strength. (It is also a base model, not a chat finetune; I see a lot of people saying it is worse, simply because they interact with it as if it was a chatbot.) [0]: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu https://paperswithcode.com/sota/multi-task-language-understa... [1]: https://github.com/open-compass/MixtralKit#comparison-with-other-models https://github.com/open-compass/MixtralKit#comparison-with-o...

brucethemoose2 3y ago

Have you played with finetunes, like Cybertron? Augmented in wrappers and retrievers like GPT is? It's not there yet, but its waaaay closer than the plain Mistral chat release.