3 ms·

It's mostly gamed by open source zealots, who have plenty of time in their hands to send "OSS, BEST" on the internet

refulgentis 3y ago
This is incorrect. This is the first time there's been a non-private model at the top, and I'd wager top 3. This isn't OSS either. Not sure where the jaundiced view comes from. I generally believe the free models tend to be overrated, but I don't think it's a good idea to pretend there's widespread gaming of this. There's a strong self-peasantization streak in LLM discussions, and truth is, no one thing may be "correct", but we can't exclude all data without making the very act of discussion meaningless.
- zooq_ai 3y ago
  You have to be seriously delusional to think that a 70B model will beat Claude Opus. But go ahead and use it to prove my point. I know the power of Zealotry and Cult in the age of internet and this is just one example of that
  - refulgentis 3y ago
    I'm at the front of the line, I maintain an app with integrations with all 4 major providers and llama.cpp with 4 built-in models. Yesterday, I was one of the, if not the, first to get it integrated and flag to model makers they had the wrong chat template embedded. That's a lengthy way of saying, contra what you imply, I have practical experience & understand this stuff intimately. I agree with your opinion re: it's unlikely LLaMA 3 70B beats all private models, but think your way of relaying it, claiming there's widespread gaming of it by advocates for open models, is obviously incorrect, and adds more confusion rather than reducing it.
    - zooq_ai 3y ago
      Why do you take it personally? Just because you are genuinely interested doesn't mean there are army of people who is sitting their with the sole purpose of gaming leaderboards. If you think that doesn't happen, I have a bridge to sell you
      - refulgentis 3y ago
        I'm not taking it personally: frankly, I'm not sure what that would mean in this context. You asked me to try it, I explained that I did, and provided context. Thank you for clarifying: to confirm, yes, I do understand your claim is a vast cabal of OSS zealots rigs LMSys's blind A/B testing. I don't think there's anything more of value I can contribute to a discussion on that topic. Have a good weekend!
        zooq_ai 3y ago
        I'm a big fan of Elon/Tesla, but I also know Elon/Tesla (a vast majority of them) is a cult. See? it's easy to have those seemingly 'conflicting' thoughts
      - jantissler 3y ago
        Then explain to us how they game it? The test is blind. You don't know which models give the answer. If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score. Genuine question, because as far as I know there is no feasible way to game this score. But if there is, I want to know.
        deleted 3y ago
        [deleted]
        extraduder_ire 3y ago
        > If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score. However that process works would be the thing to circumvent, like by getting it to disclose some letters of its name, or any value that is different between models but the same or similar for each one. I assume it doesn't provide the numerical token values in the output or it would be trivial.
  - free_bip 3y ago
    It's not beating Claude Opus. It's rank #5, tied with Bard and Claude Sonnet.
    - zooq_ai 3y ago
      It was during the time of posting, which kind of proves my point. It is gamed. A statistically random sample of users would have yielded the same results. https://twitter.com/Teknium1/status/1781328542367883765/photo/1 https://twitter.com/Teknium1/status/1781328542367883765/phot...
      - deleted 3y ago
        [deleted]
      - free_bip 3y ago
        How does that "prove" it's gamed? If anything it proves that the system does exactly what it's supposed to do: correct itself over time using more and more samples from users.

This is incorrect. This is the first time there's been a non-private model at the top, and I'd wager top 3. This isn't OSS either. Not sure where the jaundiced view comes from. I generally believe the free models tend to be overrated, but I don't think it's a good idea to pretend there's widespread gaming of this. There's a strong self-peasantization streak in LLM discussions, and truth is, no one thing may be "correct", but we can't exclude all data without making the very act of discussion meaningless.

zooq_ai 3y ago
You have to be seriously delusional to think that a 70B model will beat Claude Opus. But go ahead and use it to prove my point. I know the power of Zealotry and Cult in the age of internet and this is just one example of that
- refulgentis 3y ago
  I'm at the front of the line, I maintain an app with integrations with all 4 major providers and llama.cpp with 4 built-in models. Yesterday, I was one of the, if not the, first to get it integrated and flag to model makers they had the wrong chat template embedded. That's a lengthy way of saying, contra what you imply, I have practical experience & understand this stuff intimately. I agree with your opinion re: it's unlikely LLaMA 3 70B beats all private models, but think your way of relaying it, claiming there's widespread gaming of it by advocates for open models, is obviously incorrect, and adds more confusion rather than reducing it.
  - zooq_ai 3y ago
    Why do you take it personally? Just because you are genuinely interested doesn't mean there are army of people who is sitting their with the sole purpose of gaming leaderboards. If you think that doesn't happen, I have a bridge to sell you
    - refulgentis 3y ago
      I'm not taking it personally: frankly, I'm not sure what that would mean in this context. You asked me to try it, I explained that I did, and provided context. Thank you for clarifying: to confirm, yes, I do understand your claim is a vast cabal of OSS zealots rigs LMSys's blind A/B testing. I don't think there's anything more of value I can contribute to a discussion on that topic. Have a good weekend!
      - zooq_ai 3y ago
        I'm a big fan of Elon/Tesla, but I also know Elon/Tesla (a vast majority of them) is a cult. See? it's easy to have those seemingly 'conflicting' thoughts
    - jantissler 3y ago
      Then explain to us how they game it? The test is blind. You don't know which models give the answer. If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score. Genuine question, because as far as I know there is no feasible way to game this score. But if there is, I want to know.
      - deleted 3y ago
        [deleted]
      - extraduder_ire 3y ago
        > If you trick them into disclosing their name, it is discarded right away and doesn't count towards the score. However that process works would be the thing to circumvent, like by getting it to disclose some letters of its name, or any value that is different between models but the same or similar for each one. I assume it doesn't provide the numerical token values in the output or it would be trivial.
- free_bip 3y ago
  It's not beating Claude Opus. It's rank #5, tied with Bard and Claude Sonnet.
  - zooq_ai 3y ago
    It was during the time of posting, which kind of proves my point. It is gamed. A statistically random sample of users would have yielded the same results. https://twitter.com/Teknium1/status/1781328542367883765/photo/1 https://twitter.com/Teknium1/status/1781328542367883765/phot...
    - deleted 3y ago
      [deleted]
    - free_bip 3y ago
      How does that "prove" it's gamed? If anything it proves that the system does exactly what it's supposed to do: correct itself over time using more and more samples from users.