3 ms·

Mistral 7B,The complete Guide of the Best 7B model

32 points by gstarwd 3y ago

gstarwd 3y ago
[flagged]
- v3ss0n 3y ago
  Reporting for spam
  - brucethemoose2 3y ago
    Yeah, I almost flagged it. My spam detector was broken on this one.
brucethemoose2 3y ago
No offense, but at first glance I thought this was a GPT generated SEO ad farm... with titles like "Best AI chat bot Zephyr 7B inside finetuning Mistral 7b," "How to Finetune mistral 7B for free and easily with AutoTrain" and the general format. But its ad free, with no promotional links, and the information is good. For instance, I like the VRAM calculator, and the list of the top finetunes is excellent (like Zephyr Beta and SciPhi, both of which are very new and quite impressive). Mistral is indeed amazing. I hope they keep updating it, and maybe even publish a 13B version.
- tallytarik 3y ago
  > No offense, but at first glance I thought this was a GPT generated SEO ad farm... You're not far off. The majority of the content is ripped from the real site: https://mistral.ai/news/announcing-mistral-7b/ https://mistral.ai/news/announcing-mistral-7b/ and https://docs.mistral.ai/usage/how-to-use/ https://docs.mistral.ai/usage/how-to-use/ Unfortunately the real site doesn't have gems like "Top rated Vedio about Mistral 7B " in 30pt italics.
  - brucethemoose2 3y ago
    Its weird... it has AI generated images for the finetunes listings as well, complete with some text, with descriptions taken from HF of course. But some of it is... off. Seemingly not human curated. And of course the Mistral company page does not list any finetunes. Is this a click farm? Bait and switch for future ads? What the heck is this?
behnamoh 3y ago
IIRC there was a recent paper that showed Mistral may have data contamination, leading to high scores on the benchmarks. In my experiences with Zephyr, OpenOrca, and Dolphin finetunes of Mistral, the model did demonstrate ChatGPT-esque format in responses, but it didn't demonstrate any more intelligence than vanilla Llama 2 7B. Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.
- kcorbitt 3y ago
  I don't know whether Mistral had benchmark data contamination or not, but it's definitely a stronger model than Llama 2 7b or 13b even on non-benchmark tasks. We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
- avereveard 3y ago
  I've tested it and many fine tunes and while it seems to work ok for short chat it's really inattentive in summarisation and information extraction tasks. Continuation of long context stories/chat is also severely lacking, making up lot of contradictory details as it goes on. It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples. But it's definitely not a pick and go model like a Vicuna-13b would be.
- brucethemoose2 3y ago
  Eh, I've found llama 7b finetunes to be underwhelming too. They both seem to fall apart at detail retrieval and such, even with Mistral's longer base context. For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating https://rentry.co/ayumi_erp_rating
BrunoJo 3y ago
We are using Mistral at https://lemonfox.ai/ https://lemonfox.ai/. With Lemonfox you can easily switch from the OpenAI API to open source AI models. Mistral has been working well as a gpt-3.5 replacement but we are currently planning to switch to Zephyr - a fine-tuned Mistral model. We've seen some nice improvements with Zephyr especially with non-English prompts/responses.
- brucethemoose2 3y ago
  Are y'all doing anything in particular to stretch the context out? Just plain old rope scaling? While it seems to be a base model, I'm pretty excited about coming finetunes on this one: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k

32 points by gstarwd 3y ago

gstarwd 3y ago

[flagged]

v3ss0n 3y ago
Reporting for spam
- brucethemoose2 3y ago
  Yeah, I almost flagged it. My spam detector was broken on this one.

brucethemoose2 3y ago

No offense, but at first glance I thought this was a GPT generated SEO ad farm... with titles like "Best AI chat bot Zephyr 7B inside finetuning Mistral 7b," "How to Finetune mistral 7B for free and easily with AutoTrain" and the general format. But its ad free, with no promotional links, and the information is good. For instance, I like the VRAM calculator, and the list of the top finetunes is excellent (like Zephyr Beta and SciPhi, both of which are very new and quite impressive). Mistral is indeed amazing. I hope they keep updating it, and maybe even publish a 13B version.

tallytarik 3y ago
> No offense, but at first glance I thought this was a GPT generated SEO ad farm... You're not far off. The majority of the content is ripped from the real site: https://mistral.ai/news/announcing-mistral-7b/ https://mistral.ai/news/announcing-mistral-7b/ and https://docs.mistral.ai/usage/how-to-use/ https://docs.mistral.ai/usage/how-to-use/ Unfortunately the real site doesn't have gems like "Top rated Vedio about Mistral 7B " in 30pt italics.
- brucethemoose2 3y ago
  Its weird... it has AI generated images for the finetunes listings as well, complete with some text, with descriptions taken from HF of course. But some of it is... off. Seemingly not human curated. And of course the Mistral company page does not list any finetunes. Is this a click farm? Bait and switch for future ads? What the heck is this?

behnamoh 3y ago

IIRC there was a recent paper that showed Mistral may have data contamination, leading to high scores on the benchmarks. In my experiences with Zephyr, OpenOrca, and Dolphin finetunes of Mistral, the model did demonstrate ChatGPT-esque format in responses, but it didn't demonstrate any more intelligence than vanilla Llama 2 7B. Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.

kcorbitt 3y ago
I don't know whether Mistral had benchmark data contamination or not, but it's definitely a stronger model than Llama 2 7b or 13b even on non-benchmark tasks. We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
avereveard 3y ago
I've tested it and many fine tunes and while it seems to work ok for short chat it's really inattentive in summarisation and information extraction tasks. Continuation of long context stories/chat is also severely lacking, making up lot of contradictory details as it goes on. It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples. But it's definitely not a pick and go model like a Vicuna-13b would be.
brucethemoose2 3y ago
Eh, I've found llama 7b finetunes to be underwhelming too. They both seem to fall apart at detail retrieval and such, even with Mistral's longer base context. For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating https://rentry.co/ayumi_erp_rating

BrunoJo 3y ago

We are using Mistral at https://lemonfox.ai/ https://lemonfox.ai/. With Lemonfox you can easily switch from the OpenAI API to open source AI models. Mistral has been working well as a gpt-3.5 replacement but we are currently planning to switch to Zephyr - a fine-tuned Mistral model. We've seen some nice improvements with Zephyr especially with non-English prompts/responses.

brucethemoose2 3y ago
Are y'all doing anything in particular to stretch the context out? Just plain old rope scaling? While it seems to be a base model, I'm pretty excited about coming finetunes on this one: https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k https://huggingface.co/SciPhi/SciPhi-Mistral-7B-32k