4 ms·

Has it, though? LMSys Arena Leaderboard (blind ranking by humans) [0] positions Opus just below GPT-4 with a negligible ELO gap. [0] https://chat.lmsys.org/ ht

• BoorishBears 3y ago
  Chatbot Arena is not a blind ranking. Many, if not most, users intentionally ask the models questions to tease out their canned disclaimers: so they know exactly which model is answering. On one hand it's fair to say disclaimers affect the usefulness of the model, but on the other I don't think most people are solely asking these LLMs to produce meth or say "fuck", and that has an outsized effect on the usefulness of Chatbot Arena as a general benchmark. I personally recommend people use it at most as a way to directly test specific LLMs and ignore it as a benchmark.
• staticman2 3y ago
  That "blind ranking" is limited to about 2,000 tokens of context. So it's certainly not evaluating how good the models are at complex assignments.
• espadrine 3y ago
  A number of AI companies have a naming/reproducibility issue. GPT4 Turbo, released last November, is a separate version that is much better than GPT-4 (winning 70% of human preferences in blind tests), released in March 2023. Claude 3 Opus beats release-day GPT-4 (winning 60% of human preferences), but not GPT-4 Turbo. In the LMSys leaderboard, release-day GPT-4 is labeled gpt-4-0314, and GPT4 Turbo is labeled gpt-4-1106-preview.