5 ms·

For people who directly want to check the benchmark - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFace

• kossTKR 3y ago
  Very cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
  • valine 3y ago
    GPT-4 wasn’t nerfed. https://news.ycombinator.com/item?id=36155267 https://news.ycombinator.com/item?id=36155267
    • fbrncci 3y ago
      Very very hard to believe.
    • Taek 3y ago
      *the api wasn't nerfed. ChatGPT4 was most certainly changed
    • kossTKR 3y ago
      The comments seem to conclude that it was actually nerfed just not the API.
    • johnwheeler 3y ago
      It most certainly was, despite what anyone from openai says. It’s not working the same.
    • az226 3y ago
      It most definitely was. The chat output of GPT4 is much faster now and much worse quality. If you go in the playground and use the March 14 api (as opposed to the default GPT4 API) it is high quality and slow. Yes, there are two GPT4 API endpoints .
  • devmunchies 3y ago
    nerfed how? I didn't know this.
    • Xenoamorphous 3y ago
      Prob. referring to this https://news.ycombinator.com/item?id=36134249 https://news.ycombinator.com/item?id=36134249
    • low_tech_punk 3y ago
      maybe the comment was about safety related training leading to performance loss. Sebastien used the "Unicorn benchmark" to visualize such nerfing. Watch his talk at timestamp 26:22. Ref: https://youtu.be/qbIk7-JPB2c?t=1582 https://youtu.be/qbIk7-JPB2c?t=1582
  • redox99 3y ago
    None of these models are even close to GPT 3.5, let alone GPT4.
  • SparkyMcUnicorn 3y ago
    Here's the discussion for that benchmark with some values. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/30 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
• senttoschool 3y ago
  How does it compare to OpenAI's GPT4? It's not listed in the table.