3 ms·

Mixtral is good but those Ravenwolf benchmarks are meaningless. It’s like some random dude trying to reinvent MMLU without any rigor or consistency and in Germa

Mixtral is good but those Ravenwolf benchmarks are meaningless. It’s like some random dude trying to reinvent MMLU without any rigor or consistency and in German. Dataset contamination is a problem, but not one that’s solved by folkloric evaluation of LLMs by people asking for tips on a subreddit.

vessenes 3y ago
I don't think they're meaningless; they have a few benefits: 1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed 2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests 3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning 4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena 5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases. Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations. That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.
- bugglebeetle 3y ago
  Any tests that are unfalsifiable and can’t be reproduced are meaningless when it comes to gauging the performance of LLMs (and most other things). I could also post on a subreddit and say I have a secret set of German tests that may or may not exist and and I like these models, but that does nothing to advance the science of evaluating these things. If you want to evaluate human preferences, you can use chatbot arena, which can be gamed, but at least reflects more than what one guy says to be true. And this is with me agreeing that Nous-Capybara is also a good model. But don’t take my word for it because it’s not worth very much!
  - vessenes 3y ago
    I think we agree on almost everything about this - which is to say, probably both you and I think the Wolfram Ravenwolf tests are just about useful enough to indicate whether or not a model might be worth downloading, but certainly not enough to justify spending money, say, or planning around. So, yes, I'm with you. I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans. However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.
    - bugglebeetle 3y ago
      I think there’s room for private tests, but those should probably be done by some kind of independent standards body. LLMs are an incredible advancement of machine learning research, but we diminish that when we let these kind of wholly unscientific evaluations predominate and especially in open source, where so much great work is otherwise being done to combat the monopoly tactics of big tech.
- audessuscest 3y ago
  I agree with all except using german, because some model might be better at german but it doesn't necessarily means they are better overall. For example I'm pretty sure Mistral models are better at french, so doing a benchmark using french only would be advantageous for them. If you want to compare all models, better use english. Because now his benchmark just show which models is better at german. That being said, it's still a very welcomed benchmark.
epups 3y ago
I find it baffling that anyone would take these benchmarks seriously. The methodology is not transparent, and some of the tests completely unfair, like those in German or about specific niche German topics. The author readily acknowledges that these tests are his personal interests, which is totally fair. But that it would rise to the top of that subreddit and now HN as a general measure of quality of any kind is indicative of the lack of reliable benchmarks out there.
- bugglebeetle 3y ago
  It’s all just people jockeying for influence and hype, like crypto before it.

vessenes 3y ago

I don't think they're meaningless; they have a few benefits: 1) He doesn't have an ax to grind / an LLM to pimp out, so he's relatively even-handed 2) He uses the same (secret) test data for each model, so his testing is resistant to cherry-picking/finetuning on tests 3) He likes weirdo role-play prompting, so he has a very good sense of the edges of refusal and alignment tuning 4) He picks up stuff well before it hits the only other fair testing I know of, the chat arena 5) I think asking stuff in German is at worst neutral, and at best useful for testing capacity in edge cases. Practically speaking, his 'preferred' non-giant models, Nous-Capybara-34B and Mixtral both are excellent in comparison with some of the others he looks at, and good recommendations. That said, I'd like to see a test suite that GPT-4 fails at, or struggles at, at least. And, it would save him a lot of time if he could get something automated together, it's clearly a lot of effort to hand test all those models.

bugglebeetle 3y ago
Any tests that are unfalsifiable and can’t be reproduced are meaningless when it comes to gauging the performance of LLMs (and most other things). I could also post on a subreddit and say I have a secret set of German tests that may or may not exist and and I like these models, but that does nothing to advance the science of evaluating these things. If you want to evaluate human preferences, you can use chatbot arena, which can be gamed, but at least reflects more than what one guy says to be true. And this is with me agreeing that Nous-Capybara is also a good model. But don’t take my word for it because it’s not worth very much!
- vessenes 3y ago
  I think we agree on almost everything about this - which is to say, probably both you and I think the Wolfram Ravenwolf tests are just about useful enough to indicate whether or not a model might be worth downloading, but certainly not enough to justify spending money, say, or planning around. So, yes, I'm with you. I agree that better ways to evaluate models would be super super useful, and benchmarks like MMLU and whatever's next will continue to be helpful ("real" science). And, it seems like there may even be some benefits for models to training to 'ace the test' more broadly, which is interesting, and ties to some educational theories in teaching humans. However, one area that open tests can't excel in is in this "fair" evaluation arena -- and I do think that private tests have some value there, to the extent that they can show utility and maintain trust. I don't make any claims about German sex role-play being a good or bad start for these, though.
  - bugglebeetle 3y ago
    I think there’s room for private tests, but those should probably be done by some kind of independent standards body. LLMs are an incredible advancement of machine learning research, but we diminish that when we let these kind of wholly unscientific evaluations predominate and especially in open source, where so much great work is otherwise being done to combat the monopoly tactics of big tech.
audessuscest 3y ago
I agree with all except using german, because some model might be better at german but it doesn't necessarily means they are better overall. For example I'm pretty sure Mistral models are better at french, so doing a benchmark using french only would be advantageous for them. If you want to compare all models, better use english. Because now his benchmark just show which models is better at german. That being said, it's still a very welcomed benchmark.

epups 3y ago

I find it baffling that anyone would take these benchmarks seriously. The methodology is not transparent, and some of the tests completely unfair, like those in German or about specific niche German topics. The author readily acknowledges that these tests are his personal interests, which is totally fair. But that it would rise to the top of that subreddit and now HN as a general measure of quality of any kind is indicative of the lack of reliable benchmarks out there.

bugglebeetle 3y ago
It’s all just people jockeying for influence and hype, like crypto before it.