3 ms·

I'm a little out of touch, is there a way to see how Grok measures up to other models?

by elfbargpt 3y ago

amrrs 3y ago
Benchmarks here https://x.ai/blog/grok https://x.ai/blog/grok
- refulgentis 3y ago
  And to compare, you can sort by MMLU on here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb.... Edit: to include my self summary after review: There's a good 100 models better than, a couple 1x7b even. Mixtral stomps it, half mixtral are universally better but one is close to same.
  - lossolo 3y ago
    This benchmark is mostly worthless, some of the top models there were trained on benchmark data, which is a known fact in the community. The only reliable benchmark: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
    - refulgentis 3y ago
      No, it's not "mostly worthless" and yes, some of the top models were removed a few months back from being trained on benchmark data. I urge you to at least think through what alternative you propose before posting so aggressively in these situations. Lmsys doesn't have Grok, or I would have included it. And having _some_ data is better than none. I also had someone arguing with me 6 months back that we can't trust any benchmarks at all from vendors, which would exclude the blog post. Instead of just repeating that back vehemently, I filled a gap. It's important we don't self-peasantize as a species, all data has its issues, that doesn't mean we throw it all out.
      - michaelt 3y ago
        Quantifiable metrics are useful if they're credible, certainly. But does it seem likely, to you, that a 7B-parameter model would outperform a 314B-parameter model? Given that we can look at the chatbot arena leaderboard and it's dominated by proprietary, 70B and 8x7B models? A well regarded and modern model like Mixtral 8x7B, which is ranked 13th on the chatbot arena leaderboard, scores 72.7 'Average' on the open LLM leaderboard - and yet 'pastiche-crown-clown-7b-dare-dpo' scores 76.5. To me, that sounds too good to be true.
        refulgentis 3y ago
        Yup, 100%. Grok isn't very good and it was rushed. Rest re: pastiche model, etc. are proposing things I'm not claiming, or close to what I'm claiming. n.b. you don't multiply the parameters by experts to get an effective parameter count. Why? Think of it this way: every expert needs to learn how to speak English, so there's a nontrivial amount of duplication among all experts
        michaelt 3y ago
        > n.b. you don't multiply the parameters by experts to get an effective parameter count. I actually took the 314B from Grok's HF page [1] which describes the model as "314B parameters" when explaining why it needs a multi-GPU machine. I certainly agree that parameter count isn't everything, though; clearly things like training data quality and fine tuning count for a lot. [1] https://huggingface.co/xai-org/grok-1 https://huggingface.co/xai-org/grok-1
- salo0oh1989 3y ago
  [flagged]

I'm a little out of touch, is there a way to see how Grok measures up to other models?

amrrs 3y ago

Benchmarks here https://x.ai/blog/grok https://x.ai/blog/grok

refulgentis 3y ago
And to compare, you can sort by MMLU on here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb.... Edit: to include my self summary after review: There's a good 100 models better than, a couple 1x7b even. Mixtral stomps it, half mixtral are universally better but one is close to same.
- lossolo 3y ago
  This benchmark is mostly worthless, some of the top models there were trained on benchmark data, which is a known fact in the community. The only reliable benchmark: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
  - refulgentis 3y ago
    No, it's not "mostly worthless" and yes, some of the top models were removed a few months back from being trained on benchmark data. I urge you to at least think through what alternative you propose before posting so aggressively in these situations. Lmsys doesn't have Grok, or I would have included it. And having _some_ data is better than none. I also had someone arguing with me 6 months back that we can't trust any benchmarks at all from vendors, which would exclude the blog post. Instead of just repeating that back vehemently, I filled a gap. It's important we don't self-peasantize as a species, all data has its issues, that doesn't mean we throw it all out.
    - michaelt 3y ago
      Quantifiable metrics are useful if they're credible, certainly. But does it seem likely, to you, that a 7B-parameter model would outperform a 314B-parameter model? Given that we can look at the chatbot arena leaderboard and it's dominated by proprietary, 70B and 8x7B models? A well regarded and modern model like Mixtral 8x7B, which is ranked 13th on the chatbot arena leaderboard, scores 72.7 'Average' on the open LLM leaderboard - and yet 'pastiche-crown-clown-7b-dare-dpo' scores 76.5. To me, that sounds too good to be true.
      - refulgentis 3y ago
        Yup, 100%. Grok isn't very good and it was rushed. Rest re: pastiche model, etc. are proposing things I'm not claiming, or close to what I'm claiming. n.b. you don't multiply the parameters by experts to get an effective parameter count. Why? Think of it this way: every expert needs to learn how to speak English, so there's a nontrivial amount of duplication among all experts
        michaelt 3y ago
        > n.b. you don't multiply the parameters by experts to get an effective parameter count. I actually took the 314B from Grok's HF page [1] which describes the model as "314B parameters" when explaining why it needs a multi-GPU machine. I certainly agree that parameter count isn't everything, though; clearly things like training data quality and fine tuning count for a lot. [1] https://huggingface.co/xai-org/grok-1 https://huggingface.co/xai-org/grok-1
salo0oh1989 3y ago
[flagged]