3 ms·

I'm all for honest assessments, but this is one of the topics where I feel I have to push for constructive criticism. The LLM Arena is maybe the only comparison

I'm all for honest assessments, but this is one of the topics where I feel I have to push for constructive criticism. The LLM Arena is maybe the only comparison tool that's collectively seen as above suspicion, which is probably why it's under attack (if it actually is.) Benchmarks and tests are too easy to fit the model to, and there is a proliferation of them so they can be cherry picked. So unless we can point to something better, we can be constructive and just improve LMSYS by using it.

orbital-decay 2y ago
What if we cannot point to something better? Freeform human ranking is too easy to manipulate as each model has recognizable outputs, and publicity boost might be enough to make manipulations appealing. To be more constructive, here's a few things that make me instantly suspicious of this leaderboard in its current state. I'm talking about the API experience of course, not web frontends with unspecified amount of instructions behind the scenes. - Anything ranking above Anthropic models in non-English languages, maybe except Gemini. - 4o mini ranking above Sonnet. Seriously? It handles every task I give it worse than Sonnet, which is consistent with its purpose. - Sonnet and 4o ranking way above Opus in multiturn conversations. Every model is plagued by self-reinforced repetition in multiturn, but 4o and Sonnet are especially bad at this, the latter is nearly unusable in multiturn due to this. They're clearly not optimized for multiturn chats. While Anthropic models have terrible long context degradation and lost-in-the-middle issue, Opus is pretty reluctant to repeat itself, and can actually be used in multiturn, despite often being less performant than both. I also have plenty of questions regarding open-weights models as well.
- worstspotgain 2y ago
  Two possibilities come to mind. Either the arena rankings are being manipulated (as you seem to allege,) or there are honest queries that are fundamentally unlike your own that are tilting results away from where you'd expect. Either way, to be truly constructive, you could point to an alternative, or to a way to improve the current alternative. For example, by improving manipulation detection, or by differentiating rankings by query category.
- viraptor 2y ago
  Fwiw I haven't noticed those issues with sonnet. I ran 10+ messages deep debugging sessions and used it with Plandex for coding. Never got the impression that it gets lost. What kind of multi-turn scenarios did you experience that with?

orbital-decay 2y ago

What if we cannot point to something better? Freeform human ranking is too easy to manipulate as each model has recognizable outputs, and publicity boost might be enough to make manipulations appealing. To be more constructive, here's a few things that make me instantly suspicious of this leaderboard in its current state. I'm talking about the API experience of course, not web frontends with unspecified amount of instructions behind the scenes. - Anything ranking above Anthropic models in non-English languages, maybe except Gemini. - 4o mini ranking above Sonnet. Seriously? It handles every task I give it worse than Sonnet, which is consistent with its purpose. - Sonnet and 4o ranking way above Opus in multiturn conversations. Every model is plagued by self-reinforced repetition in multiturn, but 4o and Sonnet are especially bad at this, the latter is nearly unusable in multiturn due to this. They're clearly not optimized for multiturn chats. While Anthropic models have terrible long context degradation and lost-in-the-middle issue, Opus is pretty reluctant to repeat itself, and can actually be used in multiturn, despite often being less performant than both. I also have plenty of questions regarding open-weights models as well.

worstspotgain 2y ago
Two possibilities come to mind. Either the arena rankings are being manipulated (as you seem to allege,) or there are honest queries that are fundamentally unlike your own that are tilting results away from where you'd expect. Either way, to be truly constructive, you could point to an alternative, or to a way to improve the current alternative. For example, by improving manipulation detection, or by differentiating rankings by query category.
viraptor 2y ago
Fwiw I haven't noticed those issues with sonnet. I ran 10+ messages deep debugging sessions and used it with Plandex for coding. Never got the impression that it gets lost. What kind of multi-turn scenarios did you experience that with?