3 ms·
The lack of the comparison (which absolutely was done), tells you exactly what you need to know.
by constantcrying 10mo ago
The lack of the comparison (which absolutely was done), tells you exactly what you need to know.
- crimsoneer 10mo agoIf someone is using these models, they probably can't or won't use the existing SOTA models, so not sure how useful those comparisons actually are. "Here is a benchmark that makes us look bad from a model you can't use on a task you won't be undertaking" isn't actually helpful (and definitely not in a press release).
- constantcrying 10mo agoCompletely agree, that there are legitimate reasons to prefer comparison to e.g. deepeek models. But that doesn't change my point, we both agree that the comparisons would be extremely unfavorable.
- Lapel2742 10mo ago> that the comparisons would be extremely unfavorable. Why should they compare apples to oranges? Ministral3 Large costs ~1/10th of Sonnet 4.5. They clearly target different users. If you want a coding assistant you probably wouldn't choose this model for various reasons. There is place for more than only the benchmark king.
- constantcrying 10mo agoCome on. Do you just not read posts at all?
- esafak 10mo agoWhich lightweight models do these compare unfavorably with?
- tarruda 10mo agoHere's what I understood from the blog post: - Mistral Large 3 is comparable with the previous Deepseek release. - Ministral 3 LLMs are comparable with older open LLMs of similar sizes.
- constantcrying 10mo agoAnd implicit in this is that it compares very poorly to SOTA models. Do you disagree with that? Do you think these Models are beating SOTA and they did not include the benchmarks, because they forgot?
- tarruda 10mo ago> Do you disagree with that? I think that Qwen3 8B and 4B are SOTA for their size. The GPQA Diamond accuracy chart is weird: Both Qwen3 8B and 4B have higher scores, so they used this weid chart where "x" axis shows the number of output tokens. I missed the point of this.
- saubeidl 10mo agoThose are SOTA for open models. It's a separate league from closed models entirely.
- supermatt 10mo ago> It's a separate league from closed models entirely. To be fair, the SOTA models aren't even a single LLM these days. They are doing all manner of tool use and specialised submodel calls behind the scenes - a far cry from in-model MoE.
- popinman322 10mo agoThey're comparing against open weights models that are roughly a month away from the frontier. Likely there's an implicit open-weights political stance here. There are also plenty of reasons not to use proprietary US models for comparison: The major US models haven't been living up to their benchmarks; their releases rarely include training & architectural details; they're not terribly cost effective; they often fail to compare with non-US models; and the performance delta between model releases has plateaued. A decent number of users in r/LocalLlama have reported that they've switched back from Opus 4.5 to Sonnet 4.5 because Opus' real world performance was worse. From my vantage point it seems like trust in OpenAI, Anthropic, and Google is waning and this lack of comparison is another symptom.
- extr 10mo ago??? Closed US frontier models are vastly more effective than anything OSS right now, the reason they didn’t compare is because they’re a different weight class (and therefore product) and it’s a bit unfair. We’re actually at a unique point right now where the gap is larger than it has been in some time. Consensus since the latest batch of releases is that we haven’t found the wall yet. 5.1 Max, Opus 4.5, and G3 are absolutely astounding models and unless you have unique requirements some way down the price/perf curve I would not even look at this release (which is fine!)
- kalkin 10mo agoScale AI wrote a paper a year ago comparing various models performance on benchmarks to performance on similar but held-out questions. Generally the closed source models performed better, and Mistral came out looking pretty badly: https://arxiv.org/pdf/2405.00332 https://arxiv.org/pdf/2405.00332
- bildung 10mo agoI think people from the US often aren't aware how many companies from the EU simply won't risk losing their data to the providers you have in mind, OpenAI, Anthropic and Google. They simply are no option at all. The company I work for for example, a mid-sized tech business, currently investigates their local hosting options for LLMs. So Mistral certainly will be an option, among the Qwen familiy and Deepseek. Mistral is positioning themselves for that market, not the one you have in mind. Comparing their models with Claude etc. would mean associating themselves with the data leeches, which they probably try to avoid.
- BoorishBears 10mo agoMistral is founded by multiple Meta engineers, no? Funded mostly by US VCs? Hosted primarily on Azure? Do you really have to go out of your way to start calling their competition "data leeches" for out-executing them?
- sofixa 10mo agoMistral are mostly focusing on b2b, and for customers that want to self-host (banks and stuff). So their founders being from Meta, or where their cloud platform are hosted, are entirely irrelevant to the story.
- BoorishBears 10mo agoThe fact they would not exist without the leeches and built their business on the leeches is irrelevant. Pan-nationalism is a hell of a drug: a company that does not know you exist puts out an objectively awful release, and people take frank discussion of it as a personal slight.
- sofixa 10mo ago> The fact they would not exist without the leeches and built their business on the leeches is irrelevant. How so?
- 10mo ago