2 ms·

IIRC there was a recent paper that showed Mistral may have data contamination, leading to high scores on the benchmarks. In my experiences with Zephyr, OpenOrca

IIRC there was a recent paper that showed Mistral may have data contamination, leading to high scores on the benchmarks. In my experiences with Zephyr, OpenOrca, and Dolphin finetunes of Mistral, the model did demonstrate ChatGPT-esque format in responses, but it didn't demonstrate any more intelligence than vanilla Llama 2 7B. Another thing to worry about is that Mistral (the company) has publicly said that future larger models will not necessarily be open like the 7b model. After a certain model lock-in has taken place, the company will guard the model behind API (as they mentioned) and it won't be open anymore.

kcorbitt 3y ago
I don't know whether Mistral had benchmark data contamination or not, but it's definitely a stronger model than Llama 2 7b or 13b even on non-benchmark tasks. We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.
avereveard 3y ago
I've tested it and many fine tunes and while it seems to work ok for short chat it's really inattentive in summarisation and information extraction tasks. Continuation of long context stories/chat is also severely lacking, making up lot of contradictory details as it goes on. It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples. But it's definitely not a pick and go model like a Vicuna-13b would be.
brucethemoose2 3y ago
Eh, I've found llama 7b finetunes to be underwhelming too. They both seem to fall apart at detail retrieval and such, even with Mistral's longer base context. For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating https://rentry.co/ayumi_erp_rating
tayo42 3y ago
I wonder if it's possible to do something like folding at home but train an llm and have it always be open
- brucethemoose2 3y ago
  https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals Inference only: https://lite.koboldai.net/ https://lite.koboldai.net/
  - tayo42 3y ago
    wow didnt know that, thats really cool, ill have to play with it
    - brucethemoose2 3y ago
      If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/aphrodite-engine https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment. I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.

kcorbitt 3y ago

I don't know whether Mistral had benchmark data contamination or not, but it's definitely a stronger model than Llama 2 7b or 13b even on non-benchmark tasks. We proactively tested Mistral across all of our customer-deployed models on our fine-tuning platform (openpipe.ai) and found that with the exact same training data it outperformed both of those models consistently. At this point we're only keeping Llama 2 7b/13b around for our legacy customers and recommend that everyone training new models use Mistral because it's just so much better.

avereveard 3y ago

I've tested it and many fine tunes and while it seems to work ok for short chat it's really inattentive in summarisation and information extraction tasks. Continuation of long context stories/chat is also severely lacking, making up lot of contradictory details as it goes on. It might or might not be better in benchmarks, but I just don't see it in practice for many tasks where you'd pick a LLM and go There is one thing tho where I've seen it perform really well, which is single task fine tuning. It's small enough you can do full tuning and it's steerable enough you don't need more than few thousand samples. But it's definitely not a pick and go model like a Vicuna-13b would be.

brucethemoose2 3y ago

Eh, I've found llama 7b finetunes to be underwhelming too. They both seem to fall apart at detail retrieval and such, even with Mistral's longer base context. For what its worth, Mistral finetunes excel in this ranking, and Mistral AI almost certainly did not contaminate it with that test: https://rentry.co/ayumi_erp_rating https://rentry.co/ayumi_erp_rating

tayo42 3y ago

I wonder if it's possible to do something like folding at home but train an llm and have it always be open

brucethemoose2 3y ago
https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals Inference only: https://lite.koboldai.net/ https://lite.koboldai.net/
- tayo42 3y ago
  wow didnt know that, thats really cool, ill have to play with it
  - brucethemoose2 3y ago
    If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible: https://github.com/PygmalionAI/aphrodite-engine https://github.com/PygmalionAI/aphrodite-engine https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp Picking a model to host is quite a rabbit hole, but I am very interested in SciPhi and Zephyr at the moment. I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.