4 ms·
Mistral and its hybrids are a lot better than GPT3.5, and while not as good as GPT4 in general tasks - they’re extremely fast and powerful with specific tasks.
by smcleod 3y ago
Mistral and its hybrids are a lot better than GPT3.5, and while not as good as GPT4 in general tasks - they’re extremely fast and powerful with specific tasks. In the time it takes GPT4 to apologise that it’s not allowed to do something I can be three iterations deep getting highly targeted responses from mistral - and best yet - I can run it 100% offline, locally and on my laptop.
- stavros 3y agoAre they a lot better than 3.5? I see wildly varying opinions.
- MacsHeadroom 3y agoMistral-Medium, the one announced here which beats GPT-3.5 on every benchmark, isn't even available yet. Those opinions are referencing Mistral-Tiny (aka Mistral-7B). However, Mistral-Tiny beats the latest GPT-3.5 in human ratings on the chatbot-arena-leaderboard, in the form of OpenHermes-2.5-Mistral-7B. Mixtral 8x7B aka (Mistral-Small) was released a couple of days ago and will likely come close to GPT-4, and well above GPT-3.5, on the leaderboards once it has gone through some finetuning.
- whimsicalism 3y agoThey could be. It is an open question whether the driving force will be OSS improving or OAI continuing to try to distill their model.
- sorokod 3y agoThere is an attempt to quantify subjective evaluation of models here[1] - the "Arena Elo rating". According to popular vote, Mistral chat is nowhere near GPT 3.5 [1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- sva_ 3y agoDoesn't seem like that's the mixture of experts model in the list? Or am I blind
- fastball 3y agoIndeed, that is a previous Mistral model.
- orbital-decay 3y agoStarling-7B, OpenChat, OpenHermes in that table are Mistral-7B finetunes, and are all above the current GPT-3.5-Turbo (1106). Note how these tiny 7B models are surrounded by much larger ones. Mixtral 8x7B is not in there yet.
- pama 3y agoELO takes a while to establish. It does not sound likely that the newer GPT3.5 is that much worse than the old one that has a clear gap to all the non proprietary models. In the immediate test, GPT-3.5 clearly outshines these models.
- orbital-decay 3y ago> ELO takes a while to establish. Well, Starling-7B was published two weeks ago; GPT-3.5-turbo-0613 is more than a month old snapshot, which should probably be enough time. OpenChat and OpenHermes are about a month old as well. >It does not sound likely that the newer GPT3.5 is that much worse than the old one In fact, this version received complaints almost immediately. https://community.openai.com/t/496732 https://community.openai.com/t/496732 >In the immediate test, GPT-3.5 clearly outshines these models. It might be so, but it's not clear to me at all. I tested Starling for a bit and was really surprised that it's a 7B model, not a 70B+ one or GPT-3.5.
- whimsicalism 3y agoI disagree - lmsys score for new chatgpt has been relatively constant, and OAI is probably trying to distill the model even further.
- camillomiller 3y agoWhat laptop are you using to run which model, and what are you using for that?
- smcleod 3y agoM2 MacBook Pro, I run man different models but mistral, zephyr, deepseek. I use Ollama and LM Studio.
- amrrs 3y agoThe easiest way is to use ollama - mistral 7b, zephyr 7b, openhermes 2 are all decent models, I guess in fact openhermes 2 can do function calling. If you further want a smaller one, stablelm-zephyr 3b is a good attempt with ollama.
- YetAnotherNick 3y agoThey are not better than GPT 3.5 except for some of the public benchmarks. Also they are not faster than GPT 3.5. And they are not cheaper if you run finetuned model for specific task.
- Davidzheng 3y agoBit of a stretch to say they're a lot better when you look at evals
- berniedurfee 3y agoI was going to ask about this. So these open models are uncensored and unconstrained?