3 ms·

Mixtral ranks higher than Gemini Pro on the (subjective) Chatbot Arena Leaderboard: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggin

• jsnell 3y ago
  Presumably in the very article this HN submission is for (https://arxiv.org/pdf/2312.11444.pdf https://arxiv.org/pdf/2312.11444.pdf), table 1.
  • coder543 3y ago
    Mixtral is missing in half of the benchmarks in that paper. Hardly conclusive. It’s also common knowledge that these benchmarks have a lot of issues[0]. A good litmus test, but not a substitute for actually seeing how the models do in the real world. On the topic of “hardly conclusive” things, Gemini Pro literally told me just a few minutes ago[1] that the Avatar movies did not have humans in them. There was no funny business in the prompting. At least Mixtral knows that Avatar has humans in it. Most of Gemini Pro’s responses have been fine, but not exceptional. [0]: one random article talking about these issues: https://www.surgehq.ai//blog/hellaswag-or-hellabad-36-of-this-popular-llm-benchmark-contains-errors https://www.surgehq.ai//blog/hellaswag-or-hellabad-36-of-thi... [1]: https://i.imgur.com/En37EJD.png https://i.imgur.com/En37EJD.png