4 ms·

People love this benchmark. And it has a solid idea behind it, but it also seems like it would be trivial to game since it relies on public “blind” rankings. A

People love this benchmark. And it has a solid idea behind it, but it also seems like it would be trivial to game since it relies on public “blind” rankings. All the models need for an advantage is just to sound more personal and confident and that’ll heavily bias most users asking generic questions or having a chat. It’ll be a terrible measure for any kind of intelligence aside from sounding human-ish. It’s also good to keep in mind that the ELO scores compute the probability of winning, so a 5 point Elo difference essentially still means it’s a 49-51 chance of user preference. Not much better than a coin flip between the first 4 models, especially considering the confidence interval.

tibbar 2y ago
I thought that surely the ELO ratings were on a different scale than the usual chess ones, but nope, it uses the same factor of 400: https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc927SGUdfr6nsR?usp=sharing#scrollTo=hytEb0aXfcwm https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc.... That means we truly do have a long way to go. However, it's also possible that user questions don't have enough 'difficulty' to distinguish -- eg, if the proportion of relatively easy user questions is high, then there are fewer opportunities for a better model to distinguish itself.
- tosh 2y ago
  this.
- futureshock 2y ago
  Totally. And it’s interesting to consider what Elo a human would have, if there was some way to account for the extra latency without unblinding the test. I suspect a human control would not be a lot higher ranked. For the most part top LLMs already sound human if you prompt or fine tune them to emulate conversational style. A determined Turing tester would tell the difference most of the time I believe, but most people submitting arena ratings are not determined.
  - modeless 2y ago
    It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderboard. You could even have the people doing the voting randomly assigned to be the human sometimes. You'd know that one of the entries was a human because of the latency, but you wouldn't know which one (obviously the chatbot response would be delayed to match). I guess the problem is people would try to game the system by leaving tells in their response so you'd know that a human wrote it. You could block obvious ones but that would create a meta game of circumventing the block.
    - dns_snek 2y ago
      I think anyone who's spent any amount of time interacting with LLMs would easily tell the human response apart because LLMs have a distinguishable style of writing that's mostly consistent across interactions. Gemini, for example, loves bullet points and splits its responses into multiple distinct sections with subheadings. Furthermore, you'd have to limit the scope of questioning to topics that the human on the other side is familiar with, otherwise they won't be able to come up with an answer to questions that aren't common knowledge, while an LLM will always happily generate pages of plausible-sounding (even if incorrect) text. I think a more appropriate benchmark would be subject matter experts interviewing an LLM and another subject matter expert, with clear guidelines regarding the style of writing they're expected to match.

tibbar 2y ago

I thought that surely the ELO ratings were on a different scale than the usual chess ones, but nope, it uses the same factor of 400: https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc927SGUdfr6nsR?usp=sharing#scrollTo=hytEb0aXfcwm https://colab.research.google.com/drive/1RAWb22-PFNI-X1gPVzc.... That means we truly do have a long way to go. However, it's also possible that user questions don't have enough 'difficulty' to distinguish -- eg, if the proportion of relatively easy user questions is high, then there are fewer opportunities for a better model to distinguish itself.

tosh 2y ago
this.
futureshock 2y ago
Totally. And it’s interesting to consider what Elo a human would have, if there was some way to account for the extra latency without unblinding the test. I suspect a human control would not be a lot higher ranked. For the most part top LLMs already sound human if you prompt or fine tune them to emulate conversational style. A determined Turing tester would tell the difference most of the time I believe, but most people submitting arena ratings are not determined.
- modeless 2y ago
  It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderboard. You could even have the people doing the voting randomly assigned to be the human sometimes. You'd know that one of the entries was a human because of the latency, but you wouldn't know which one (obviously the chatbot response would be delayed to match). I guess the problem is people would try to game the system by leaving tells in their response so you'd know that a human wrote it. You could block obvious ones but that would create a meta game of circumventing the block.
  - dns_snek 2y ago
    I think anyone who's spent any amount of time interacting with LLMs would easily tell the human response apart because LLMs have a distinguishable style of writing that's mostly consistent across interactions. Gemini, for example, loves bullet points and splits its responses into multiple distinct sections with subheadings. Furthermore, you'd have to limit the scope of questioning to topics that the human on the other side is familiar with, otherwise they won't be able to come up with an answer to questions that aren't common knowledge, while an LLM will always happily generate pages of plausible-sounding (even if incorrect) text. I think a more appropriate benchmark would be subject matter experts interviewing an LLM and another subject matter expert, with clear guidelines regarding the style of writing they're expected to match.