4 ms·
Mistral's models outperform all of theirs at much smaller sizes and can be self-hosted.
by xvector 3y ago
Mistral's models outperform all of theirs at much smaller sizes and can be self-hosted.
- mrkramer 3y agoRight now, but how long does it take for OpenAI, Facebook and Google's Bard/Gemini to catch up?
- capableweb 3y agoConsidering Google been lying in their marketing videos about their AI capabilities, OpenAI seems to be falling apart from the inside and Facebook refuses to release open (not commercially-locked "open") models, there seems to be space for at least one honest and open AI company.
- baq 3y agoGoogle is trying and just can't? They are pretending they aren't behind despite setting a pile of cash on fire by spending exorbitant amounts on exaggerating their capabilities in basically fake videos. Meanwhile you see Mistral casually dropping magnet links to weights with barely any instructions on how to use them on a Friday afternoon and you see reports of some very happy people finetuning on a civilian 4090 and getting GPT-4-quality performance in blind tests on Saturday morning.
- sailingparrot 3y agoGoogle is trying to catch up with OpenAI, not with open source 7B models.
- MacsHeadroom 3y agoOpen Source 7B models have already caught up with OpenAI. The current top of the open model 7B leaderboard beats GPT-3.5 and Bard while running on a laptop, smartphone, raspberry pi, or 10 year old graphics card. Tim Dettmers just released code to get Mixtral 8x7b running in 4GB of RAM, the same amount required by Mistral 7b. We are quite clearly a matter of weeks from Open Source being on par with GPT-4, likely before Google's Gemini Ultra is even released. This is all to say nothing of the multi-model LLama-3 120B coming in 2 months according to insider leaks.
- wenc 3y agoI’m very skeptical of LLM benchmarks because I’ve been testing models through Ollama and none of the open source 7b models are at even close to GPT4 (or Open AI’s GPT 3.5 implementation) in daily usage, despite what benchmarks say. LLM results are actually really hard to measure objectively because it depends on what kinds of questions you ask and what answers you expect. I’ve just been finding lots of little errors and reasoning mistakes, and even in terms of producing useful results, they fall short of GPT4.
- wenc 3y ago> Mistral's models outperform all of theirs at much smaller sizes and can be self-hosted. This does not track with my experience. Mistral's models have bang for buck in that they are impressive for being so small, but they do not beat GPT4 at all.
- xvector 3y agoSure, but you can't host GPT-4.
- wenc 3y agoNo, but I look forward to the day I can trust Mistral results. Right now I have low confidence in them for any use that require at least a reasonable level of correctness. GPT4 on the other hand is correct enough for many use cases and much faster. I’m also excited to see how new non transformer architectures like Mamba and state space models turn out.