3 ms·

I wished the charts included Qwen3, the current SOTA in reasoning. Qwen3-4B almost beats Magistral-22B on the 4 available benchmarks, and Qwen3-30B-A3B is mile

I wished the charts included Qwen3, the current SOTA in reasoning. Qwen3-4B almost beats Magistral-22B on the 4 available benchmarks, and Qwen3-30B-A3B is miles ahead.

resource_waste 1y ago
No surprise on my end. Mistral has been basically useless due to other models always being better. But its European, so its a point of pride. Relevance or not, we will keep hearing the name as a result.
SparkyMcUnicorn 1y ago
30-A3B is a really impressive model. I throw tasks at it running locally to save on API costs, and it's possibly better than anything we had a year or so ago from closed source providers. For programming tasks, I'd rank it higher than gpt-4o
- iamnotagenius 1y ago
  [dead]
- freehorse 1y ago
  It is a great model, and blazing fast, which is actually very useful esp for "reasoning" models, as they produce a lot of tokens. I wish mistral were back into making MoE models. I loved their 8x7 mixtral, it was one of the greatest models I could run the time it went out, but it is outdated now. I wish somebody was out making a similar size MoE model, which could comfortably sit in a 64GB ram macbook and be fast. Currently the qwen 30-A3B is the only one I know of, but it would be nice to have something slightly bigger/better (incl a non-reasoning base one). All the other MoE models are just too big to run locally in more standard hardware.
poorman 1y ago
Is there a popular benchmark site people use? Becaues I had to test all these by hand and `Qwen3-30B-A3B` still seems like the best model I can run in that relative parameter space (/memory requirements).
- arnaudsm 1y ago
  - https://livebench.ai/#/ https://livebench.ai/#/ + AIME + LiveCodeBench for reasoning - MMLU-Pro for knowledge - https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard for user preference We only got Magistral's GPQA, AIME & livecodebench so far.
devmor 1y ago
I would agree, Qwen3 is definitely the most impressive "reasoning" model I've evaluated so far.

I wished the charts included Qwen3, the current SOTA in reasoning. Qwen3-4B almost beats Magistral-22B on the 4 available benchmarks, and Qwen3-30B-A3B is miles ahead.

resource_waste 1y ago

No surprise on my end. Mistral has been basically useless due to other models always being better. But its European, so its a point of pride. Relevance or not, we will keep hearing the name as a result.

SparkyMcUnicorn 1y ago

30-A3B is a really impressive model. I throw tasks at it running locally to save on API costs, and it's possibly better than anything we had a year or so ago from closed source providers. For programming tasks, I'd rank it higher than gpt-4o

iamnotagenius 1y ago
[dead]
freehorse 1y ago
It is a great model, and blazing fast, which is actually very useful esp for "reasoning" models, as they produce a lot of tokens. I wish mistral were back into making MoE models. I loved their 8x7 mixtral, it was one of the greatest models I could run the time it went out, but it is outdated now. I wish somebody was out making a similar size MoE model, which could comfortably sit in a 64GB ram macbook and be fast. Currently the qwen 30-A3B is the only one I know of, but it would be nice to have something slightly bigger/better (incl a non-reasoning base one). All the other MoE models are just too big to run locally in more standard hardware.

poorman 1y ago

Is there a popular benchmark site people use? Becaues I had to test all these by hand and `Qwen3-30B-A3B` still seems like the best model I can run in that relative parameter space (/memory requirements).

arnaudsm 1y ago
- https://livebench.ai/#/ https://livebench.ai/#/ + AIME + LiveCodeBench for reasoning - MMLU-Pro for knowledge - https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard for user preference We only got Magistral's GPQA, AIME & livecodebench so far.

devmor 1y ago

I would agree, Qwen3 is definitely the most impressive "reasoning" model I've evaluated so far.