3 ms·
It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderbo
by modeless 3y ago
It would be interesting to have a couple of "time-limited human" entrants. 10 second human, 1 minute human, 10 minute human. See where they fall on the leaderboard. You could even have the people doing the voting randomly assigned to be the human sometimes.
You'd know that one of the entries was a human because of the latency, but you wouldn't know which one (obviously the chatbot response would be delayed to match).
I guess the problem is people would try to game the system by leaving tells in their response so you'd know that a human wrote it. You could block obvious ones but that would create a meta game of circumventing the block.
- dns_snek 3y agoI think anyone who's spent any amount of time interacting with LLMs would easily tell the human response apart because LLMs have a distinguishable style of writing that's mostly consistent across interactions. Gemini, for example, loves bullet points and splits its responses into multiple distinct sections with subheadings. Furthermore, you'd have to limit the scope of questioning to topics that the human on the other side is familiar with, otherwise they won't be able to come up with an answer to questions that aren't common knowledge, while an LLM will always happily generate pages of plausible-sounding (even if incorrect) text. I think a more appropriate benchmark would be subject matter experts interviewing an LLM and another subject matter expert, with clear guidelines regarding the style of writing they're expected to match.