11 ms·
Un Ministral, Des Ministraux
- zurfer 2y agopoor title, Mistral released new open weight models that win across benchmarks in their weight class: Ministral 3B and Ministral 8B
- scjody 2y agoAre they really open weights? Ministral 3B is "Mistral Commercial License".
- leetharris 2y agoYeah the 3B are NOT open. The 8B are as they can be used under commercial license.
- DreamGen 2y agoThat would be misleading. They aren't open weight (3B is not available). They aren't compared to Qwen 2.5 which beats them in many of the benchmarks presented while having more permissive license. The closed 3B is not competitive with other API only models, like Gemini Flash 8B which costs less and has better performance.
- WiSaGaN 2y ago"For self-deployed use, please reach out to us for commercial licenses. We will also assist you in lossless quantization of the models for your specific use-cases to derive maximum performance. The model weights for Ministral 8B Instruct are available for research use. Both models will be available from our cloud partners shortly."
- deleted 2y ago[deleted]
- lairv 2y agoHard to see how can Mistral compete with Meta, they have order of magnitude less compute, their models are only slightly better (at least on the benchmarks) with less permissive licenses?
- simonw 2y agoYeah, the license thing is definitely a problem. It's hard to get excited about an academic research license for a 3B or 8B model when the Llama 3.1 and 3.2 models are SO good, and are licensed for commercial usage.
- harisec 2y agoQwen 2.5 models are better than Llama and Mistral.
- speedgoose 2y agoI disagree. I tried the small ones but they too frequently output Chinese when the prompt is English.
- harisec 2y agoI never had this problem but i guess it depends on the prompt.
- sigmar 2y agoto be clear- these ministal model are also licensed for commercial use, but not freely licensed for commercial use. and meta also has restrictions on commercial use (have to put “Built with Meta Llama 3” and need to pay meta if you exceed 700 million monthly users)
- sthatipamala 2y agoYou need to pay meta if you have 700 million users as of the Llama 3 release date. Not at any time going forward.
- druskacik 2y agoHow many Mistral puns are there? The benchmarks look promising, great job, Mistral.
- xnx 2y agoHas anyone put together a good and regularly updated decision tree for what model to use in different circumstances (VRAM limitations, relative strengths, licensing, etc.)? Given the enormous zoo of models in circulation, there must be certain models that are totally obsolete.
- leetharris 2y agoPeople keep making these, but they become outdated so fast and nobody keeps up with it. If your definition of "great" changes in 6 months because a new model shatters your perception of "great," it's hard to rescore legacy models. I'd say keeping up with the reddit LocalLLama community is the "easiest" way and it's by no means easy.
- potatoman22 2y agoSomeone should use an LLM to continuously maintain this decision tree. The tree itself will decide which LLM is used for maintainence.
- kergonath 2y ago> I'd say keeping up with the reddit LocalLLama community is the "easiest" way and it's by no means easy. The subreddit is… not great. It’s a decent way of keeping up, but don’t read the posts too much (and even then, there is a heavy social aspect, and the models that are discussed there are a very specific subset of what’s available). There is a lot of groupthink, the discussions are never rigorous. Most of the posts are along the lines of “I tested a benchmark and it is 0.5 points ahead of Llama-whatever on that one benchmark I made up, therefore it’s the dog’s and everything else is shite”. The Zuckerberg worshiping is also disconcerting. Returns diminish quickly as you spend more time on that subreddit.
- iamjackg 2y agoThis is definitely a problem. I mostly take a look at the various leaderboards, but there is a proliferation of fine-tuned models that makes it incredibly daunting to explore the model space. Add to that that often they're not immediately available on turn-key tools like ollama, and the friction increases even more. All this without even considering things like licenses, what kind of data has been used for fine tuning, quantization, merges, multimodal capabilities. I would love a curated list.
- cmehdy 2y agoFor anybody wondering about the title, that's a sort-of pun in French about how words get pluralized following French rules. The quintessential example is "cheval" (horse) which becomes "chevaux" (horses), which is the rule they're following (or being cute about). Un mistral, des mistraux. Un ministral, des ministraux. (Ironically the plural of the Mistral wind in the Larousse dictionnary would technically be Mistrals[1][2], however weird that sounds to my french ears and to the people who wrote that article perhaps!) [1] https://www.larousse.fr/dictionnaires/francais/mistral_mistrals/51794 https://www.larousse.fr/dictionnaires/francais/mistral_mistr... [2] https://fr.wiktionary.org/wiki/mistral https://fr.wiktionary.org/wiki/mistral
- BafS 2y agoIt's complex because french is full of exceptions the classical way to pluralize "–al" words: un animal → des animaux [en: animal(s)] un journal → des journaux [en: journal(s)] with some exceptions: un carnaval → des carnavals [en: carnival(s)] un festival → des festivals [en: festival(s)] un idéal → des idéals (OR des idéaux) [en: ideal(s)] un val → des vals (OR des vaux) [en: valley(s)] There is no logic there (as many things in french), it's up to Mistral to choose how the plural can be EDIT: Format + better examples
- rich_sasha 2y agoThat's news to me that French for "valley" is masculine and "val" - isn't it feminine "vallée"? Like, say "Vallée Blanche" near Chamonix? And I suppose the English ripoff, "valley" sounds more like "vallée" than "val" (backwards argument, I know).
- mytailorisrich 2y agoYes, la vallée (feminine) and le val (masculine). Valley is usually la vallée. Val is mostly only used in the names of places. Apparently val gave vale in English.
- makapuf 2y ago
- barbegal 2y agoDoes anyone know why Mistral use a 17 bit (131k) vocabulary? I'm sure it's more efficient at encoding text but each token doesn't fit into a 16 bit register which must make it more inefficient computationally?
- cpldcpu 2y agoThe tokens are immediately transformed into embeddings (very large vectors), so the 17 bit values are not used for any computation.
- sharkjacobs 2y agoI know that Mistral is a French company, but I think it's really clever marketing the way they're using French language as branding.
- ed 2y ago3b is is API-only so you won’t be able to run it on-device, which is the killer app for these smaller edge models. I’m not opposed to licensing but “email us for a license” is a bad sign for indie developers, in my experience. 8b weights are here https://huggingface.co/mistralai/Ministral-8B-Instruct-2410 https://huggingface.co/mistralai/Ministral-8B-Instruct-2410 Commercial entities aren’t permitted to use or distribute 8b weights - from the agreement (which states research purposes only): "Research Purposes": means any use of a Mistral Model, Derivative, or Output that is solely for (a) personal, scientific or academic research, and (b) for non-profit and non-commercial purposes, and not directly or indirectly connected to any commercial activities or business operations. For illustration purposes, Research Purposes does not include (1) any usage of the Mistral Model, Derivative or Output by individuals or contractors employed in or engaged by companies in the context of (a) their daily tasks, or (b) any activity (including but not limited to any testing or proof-of-concept) that is intended to generate revenue, nor (2) any Distribution by a commercial entity of the Mistral Model, Derivative or Output whether in return for payment or free of charge, in any medium or form, including but not limited to through a hosted or managed service (e.g. SaaS, cloud instances, etc.), or behind a software layer.
- mark_l_watson 2y agoYou are correct, convenience for trying many new models is important. For me, this means being able to run with Ollama.
- diggan 2y ago> I’m not opposed to licensing but “email us for a license” is a bad sign for indie developers, in my experience. At least they're not claiming it's Open Source / Open Weights, kind of happy about that, as other companies didn't get the memo that lying/misleading about stuff like that is bad.
- talldayo 2y agoYeah, a real silver-lining on the API-only access for a model that is intentionally designed for edge devices. As a user I honestly only care about the weights being open - I'm not going to reimpliment their training code and I don't need or want redistributed training data that both already exists elsewhere. There is no benefit, for my uses, to having an "open source" model when I could have weights and finetunes instead. There's nothing to be happy about when businesses try to wall-off a feature to make you salivate over it more. You're within your right to nitpick licensing differences, but unless everyone gets government-subsidized H100s in their garage I don't think the code will be of use to anyone except moneyed competitors that want to undermine foundational work.
- gunalx 2y agoNot having open ish weigths is a total dealbreaker for me. The only really compelling reason behind sub 6B models is them being easy to run on even consumer hardware or on the edge.
- throwaway796857 2y ago[dead]
- aabhay 2y agoThis press release is a big change in branding and ethos for Mistral. What was originally a vibey, insurgent contender that put out magnet links is now a PR-crafting team that has to fight to pitch their utility to the public.
- littlestymaar 2y agoI was going to say the same. Incredible to see how quickly Mistral went from “magnet links casually dropped on twitter by their CTO” to “PR blog post without the model weights” in just a year. Not a good sign at all as it means their investors are already getting nervous.
- wg0 2y agoThat's usually the evidence of VCs getting involved. Somber corporate tone proud on accomplishments user will find useful we continue to improve looking to the future and such.
- swyx 2y agojust want to point out that this isnt entirely true. pixtral was magnet link dropped recently. mistral simply has two model rollout channels depending on the level of openness they choose. dont extrapolate too much due to vc hate.
- Whiteshadow12 2y agoNice voice of reason swyx, people who are not hooked on X, will have selective memory, "Mistral has changed, I miss the old Mistral". Last year Mistral watched as every provider host their models with little to no value capture. Nemo is Apache 2.0 license, they could have easily made that a Mistral Research License model. It's hard to pitch vc for more money to build more models when you don't capture anything making it Apache 2.0. Not everyone can be Meta. Magnet links are cute but honestly, most people rather use HF to get their models.
- csomar 2y agoThis might suggest that they are plateauing. If you think your next model won't improve a lot, then you'll try to start earning from this current model. Luckily, we still have meta. llma3.2 is really good. and it runs on my laptop with a regular intel CPU.
- tarruda 2y agoThey didn't add a comparison to Qwen 2.5 3b, which seems to surpass Ministral 3b MMLU, HumanEval, GSM8K: https://qwen2.org/qwen2-5/#qwen25-05b15b3b-performance https://qwen2.org/qwen2-5/#qwen25-05b15b3b-performance These benchmarks don't really matter that much, but it is funny how this blog post conveniently forgot to compare with a model that already exists and performs better.
- butterfly42069 2y agoAt this point the benchmarks barely matter at all. It's entirely possible to train for a high benchmark score and reduce the overall quality of the model in the process. Imo use the model that makes the most sense when you ask it stuff, and personally I'd go for the one with the least censorship (which imo isn't AliBaba Qwen anything)
- DreamGen 2y agoAlso, the 3B model, which is API only (so the only thing that matters is price, quality and speed) should be compared to something like Gemini Flash 1.5 8B which is cheaper than this 3B API and also has higher benchmark performance, super long context support, etc.
- smcleod 2y agoIt's pretty hard to claim it's the world's best then not compare it to Qwen 2.5....
- daghamm 2y agoYeah, qwen does great in benchmarks but is it really that good in real use?
- deleted 2y ago[deleted]
- akvadrako 2y agoIn my experience using it for story telling, no. Even their largest model likes to produce garbage which isn't even words when you turn the temperature up a bit. And without that it's really bland.
- smcleod 2y agoYes it's fantastic, really great for both general use and coding.
- daghamm 2y agoI don't get it. These are impressive numbers. But while their use case is local execution to preserve privacy, the only way to use these models right now is to use their API?
- amelius 2y agoI have no idea how to compare the abilities of these models, so I have no idea how much of a deal this is.
- mergisi 2y agoJust started experimenting with Ministral 8B! It even passed the "strawberry test"! https://x.com/mustafaergisi/status/1846861559059902777 https://x.com/mustafaergisi/status/1846861559059902777
- fhdsgbbcaA 2y agoThat just means it was trained recently enough to have it in training data.