3 ms·

One such project is RWKV[1]. On the open source leaderboard it lived in the middle of the board for a while, so it really is a legit approach, it's just not hot

• swyx 3y ago
  side note - do you think the open source leaderboard is a fair representation of the diversity of OSS models?
  • modeless 3y ago
    It's the best we have for large scale comparison I think. But the major problem is that the tests are public and you can easily cheat by including them in your training set and increase your score an arbitrary amount. You might even be able to do it in a way that would be difficult to detect. And it can happen by accident as well. There are already many cases of intentional and disclosed contamination in the leaderboard. What's really needed is a leaderboard based on some private test sets, where you can submit a model to be judged and some entity will run it on their own machines without disclosing the tests to you. Even that could be vulnerable to cheating if the submission process is automated and you can submit many times, as you could use the score as feedback to disclose information about the contents of the private test set. So it would take some care to run such a service, even beyond normal security concerns like securing the sandbox that submitted code runs in, etc.
    • xpe 3y ago
      Who would you trust (and verify) to run such a leaderboard?
      • modeless 3y ago
        Any of the big labs I guess. At least as long as the leaderboard didn't become something so important that even the big labs would be incentivized to cheat on it. In that case, I don't know.