4 ms·

A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to ge

by sottol 2mo ago

A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted? What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?

Iolaum 2mo ago
Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.
- logicchains 2mo ago
  It's not overthinking, it's the right amount of thinking necessary for such a small model to get good results. The dumber the model, the more it has to think to be smart. There's no easy way to reduce the thinking without reducing the model quality.
  - zdragnar 2mo ago
    Qwen doom loops were amusing to watch the first time or two, but it's incredibly vexing to have it waffle over the same decision over and over and over and over again. I can get more done with a faster model by correcting it, and it feels better to babysit them than it does to babysit qwen to see if I need to intervene or if it will actually finish. I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.
    - skohan 2mo ago
      It's only been a couple days, but I haven't seen looping issues with 3.8 so far, compared to 3.6 which did occasionally have this problem.
re5i5tor 2mo ago
Have you tried it? I’d recommend doing so, it’s impressive in real use cases.
aqme28 2mo ago
Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.
- deaux 2mo ago
  Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark. [0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deepseek-v4-flash%2Cdeepseek-v4-pro%2Cgemma-4-31b%2Cgemma-4-26b-a4b%2Cqwen3-8-27b%2Cgpt-5-6-luna https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...
  - throwa356262 2mo ago
    I dont think this is benchmaxing. They have simply decided to not train the model in some areas such as world physics
- achrono 2mo ago
  Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them. I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.
  - skohan 2mo ago
    At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
deaux 2mo ago
This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.
- culi 2mo ago
  This website aggregates 82 different benchmarks. It seems to do pretty consistently well across the ones its been tested on https://benchlm.ai/models/qwen3-5-27b#ledger https://benchlm.ai/models/qwen3-5-27b#ledger
drob518 2mo ago
IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ https://www.intelligence.ai/ http://arena.ai http://arena.ai

by sottol 2mo ago

Iolaum 2mo ago

Yea and we are reaching the point where this benchmaxing is visible in the model's reported overthinking.

logicchains 2mo ago
It's not overthinking, it's the right amount of thinking necessary for such a small model to get good results. The dumber the model, the more it has to think to be smart. There's no easy way to reduce the thinking without reducing the model quality.
- zdragnar 2mo ago
  Qwen doom loops were amusing to watch the first time or two, but it's incredibly vexing to have it waffle over the same decision over and over and over and over again. I can get more done with a faster model by correcting it, and it feels better to babysit them than it does to babysit qwen to see if I need to intervene or if it will actually finish. I do like the output from qwen when I get it, but honestly I haven't been impressed enough with it to put up with the downsides.
  - skohan 2mo ago
    It's only been a couple days, but I haven't seen looping issues with 3.8 so far, compared to 3.6 which did occasionally have this problem.

re5i5tor 2mo ago

Have you tried it? I’d recommend doing so, it’s impressive in real use cases.

aqme28 2mo ago

Do you have any evidence that this model is bench-maxxed? I know that's particularly difficult to quantify. If there is an indicator of bench-maxxing, that just becomes the new benchmark to benchmax.

deaux 2mo ago
Here, filtered down for you. [0] Look at the individual benchmarks, not the combined one. You can tell that this model is much more benchmaxxed as its relative ranking swings between benchmarks is much larger. This is a hallmark. [0] https://artificialanalysis.ai/models/qwen3-8-27b?models=deepseek-v4-flash%2Cdeepseek-v4-pro%2Cgemma-4-31b%2Cgemma-4-26b-a4b%2Cqwen3-8-27b%2Cgpt-5-6-luna https://artificialanalysis.ai/models/qwen3-8-27b?models=deep...
- throwa356262 2mo ago
  I dont think this is benchmaxing. They have simply decided to not train the model in some areas such as world physics
achrono 2mo ago
Sounds obvious but just try using the models for anything outside the evals. Take something arcane from Greek history, use it to create a masked linguistic puzzle, which you then ask the model to solve mathematically, all wrapped as an ask to generate ASCII art. Yes, all these elements exist in some form in the evals but the key is in how utterly unconventional the elements are that you pick and in how you combine them. I have consistently noticed Opus 4.8 and GPT-5.6 far outshine the Chinese models. Gemini is sort of middle of the road, Grok is better than Gemini but not really close to Opus/GPT. OAI & Anthropic still remain unbeaten by a wide margin in my eyes.
- skohan 2mo ago
  At that point aren't you just edge-case testing? Surely most of your use-cases are not novel tasks that combine obscure domains. It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.

deaux 2mo ago

This one is very benchmaxxed, and you can tell from this page alone. Look at the huge variance in ranking per benchmark. Most models, including at that size, are much more consistent.

culi 2mo ago
This website aggregates 82 different benchmarks. It seems to do pretty consistently well across the ones its been tested on https://benchlm.ai/models/qwen3-5-27b#ledger https://benchlm.ai/models/qwen3-5-27b#ledger

drob518 2mo ago

IMO, ELO rating from The Intelligence company and arena.ai are more representative of rankings since they use humans to judge a head-to-head comparison between a couple models at a time. https://www.intelligence.ai/ https://www.intelligence.ai/ http://arena.ai http://arena.ai