5 ms·

> Keep in mind that modern quantitative approaches to LLM evaluation have been effectively co-designed with the rise of OpenAI, and folks like Ravenwolf routine

by epups 3y ago

> Keep in mind that modern quantitative approaches to LLM evaluation have been effectively co-designed with the rise of OpenAI, and folks like Ravenwolf routinely disagree with the leaderboards. Sorry but you're talking complete nonsense here. The benchmark by LMSys (chatbot arena) cannot be gamed, and Ravenwolf is a random-ass poster with no scientific rigor to his benchmarks.

ParetoOptimal 3y ago
Cannot be gamed? C'mon now... You could pay a bunch of people to vote for your model in the arena.
- epups 3y ago
  No you can't, because you actually don't know which model is which when you vote.
  - ParetoOptimal 3y ago
    Do only the initial votes count? Because after I made an initial choice I was then put in a session where I saw the name of both of the AI. I made subsequent votes in that session where I could see their names.
    - epups 3y ago
      https://github.com/lm-sys/FastChat/issues/1210 https://github.com/lm-sys/FastChat/issues/1210
- deleted 3y ago
  [deleted]