4 ms·

Totally. And it’s interesting to consider what Elo a human would have, if there was some way to account for the extra latency without unblinding the test. I sus

Totally. And it’s interesting to consider what Elo a human would have, if there was some way to account for the extra latency without unblinding the test. I suspect a human control would not be a lot higher ranked. For the most part top LLMs already sound human if you prompt or fine tune them to emulate conversational style. A determined Turing tester would tell the difference most of the time I believe, but most people submitting arena ratings are not determined.

modeless 2y ago
It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderboard. You could even have the people doing the voting randomly assigned to be the human sometimes. You'd know that one of the entries was a human because of the latency, but you wouldn't know which one (obviously the chatbot response would be delayed to match). I guess the problem is people would try to game the system by leaving tells in their response so you'd know that a human wrote it. You could block obvious ones but that would create a meta game of circumventing the block.
- dns_snek 2y ago
  I think anyone who's spent any amount of time interacting with LLMs would easily tell the human response apart because LLMs have a distinguishable style of writing that's mostly consistent across interactions. Gemini, for example, loves bullet points and splits its responses into multiple distinct sections with subheadings. Furthermore, you'd have to limit the scope of questioning to topics that the human on the other side is familiar with, otherwise they won't be able to come up with an answer to questions that aren't common knowledge, while an LLM will always happily generate pages of plausible-sounding (even if incorrect) text. I think a more appropriate benchmark would be subject matter experts interviewing an LLM and another subject matter expert, with clear guidelines regarding the style of writing they're expected to match.

modeless 2y ago

It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderboard. You could even have the people doing the voting randomly assigned to be the human sometimes. You'd know that one of the entries was a human because of the latency, but you wouldn't know which one (obviously the chatbot response would be delayed to match). I guess the problem is people would try to game the system by leaving tells in their response so you'd know that a human wrote it. You could block obvious ones but that would create a meta game of circumventing the block.

dns_snek 2y ago
I think anyone who's spent any amount of time interacting with LLMs would easily tell the human response apart because LLMs have a distinguishable style of writing that's mostly consistent across interactions. Gemini, for example, loves bullet points and splits its responses into multiple distinct sections with subheadings. Furthermore, you'd have to limit the scope of questioning to topics that the human on the other side is familiar with, otherwise they won't be able to come up with an answer to questions that aren't common knowledge, while an LLM will always happily generate pages of plausible-sounding (even if incorrect) text. I think a more appropriate benchmark would be subject matter experts interviewing an LLM and another subject matter expert, with clear guidelines regarding the style of writing they're expected to match.