3 ms·

I'm struggling to figure out what to use this for. From the intelligence benchmarks in OMLX. If only they would release another MoE model. Intelligence Bench

I'm struggling to figure out what to use this for. From the intelligence benchmarks in OMLX. If only they would release another MoE model. Intelligence Benchmark Comparison --- Detail --- Model: scottlowry--Qwen3.8-27B-oQ4e-mtp Benchmark Accuracy Correct Total Time(s) Think -------------------------------------------------------------- GSM8K 93.3% 28 30 282 No MATHQA 46.7% 14 30 26.3 No HUMANEVAL 96.7% 29 30 156.5 No MBPP 83.3% 25 30 71.5 No LIVECODEBENCH 43.3% 13 30 1040.4 No Model: stamsam--Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-MLX-oQ4-MTP Benchmark Accuracy Correct Total Time(s) Think -------------------------------------------------------------- GSM8K 96.7% 29 30 51.9 No MATHQA 60.0% 18 30 9.1 No HUMANEVAL 83.3% 25 30 82.9 No MBPP 80.0% 24 30 29.6 No LIVECODEBENCH 36.7% 11 30 283.7 No

walrus01 2mo ago
Try manually asking both more discrete esoteric knowledge questions. Or use benchmarks which are less coding focused. The 3.6-35B-A3B with post-training may do well in coding type benchmarks and math but the density of its knowledge falls off in my experience (vs 3.6 27B dense Q8-K-XL unsloth GGUF) when you need to use it for less commonly used domains of knowledge.
- seanmcdirmid 2mo ago
  My use cases try to avoid accessing world knowledge in the model (I give it access to web search for some adhoc RAG), and ya, I'm just focused on coding so that's the only place I'm looking at right now.
  - walrus01 2mo ago
    I think you may find that the dense 27B also does better if challenged with more rare coding tasks, less common or weird languages or things that aren't well represented in the active 3B parameters of the MoE model (eg: NOT css, javascript, python, c++, etc).
    - seanmcdirmid 2mo ago
      True. Benchmarks have led me astray before, I'll have to actually benchmark it directly in Goose with my real use cases right now (which are mostly writing python code, so it probably still doesn't apply?). Still, I've never noticed a benefit to using a dense model rather than an MoE for coding with an agent tasks.

walrus01 2mo ago

Try manually asking both more discrete esoteric knowledge questions. Or use benchmarks which are less coding focused. The 3.6-35B-A3B with post-training may do well in coding type benchmarks and math but the density of its knowledge falls off in my experience (vs 3.6 27B dense Q8-K-XL unsloth GGUF) when you need to use it for less commonly used domains of knowledge.

seanmcdirmid 2mo ago
My use cases try to avoid accessing world knowledge in the model (I give it access to web search for some adhoc RAG), and ya, I'm just focused on coding so that's the only place I'm looking at right now.
- walrus01 2mo ago
  I think you may find that the dense 27B also does better if challenged with more rare coding tasks, less common or weird languages or things that aren't well represented in the active 3B parameters of the MoE model (eg: NOT css, javascript, python, c++, etc).
  - seanmcdirmid 2mo ago
    True. Benchmarks have led me astray before, I'll have to actually benchmark it directly in Goose with my real use cases right now (which are mostly writing python code, so it probably still doesn't apply?). Still, I've never noticed a benefit to using a dense model rather than an MoE for coding with an agent tasks.