4 ms·

this leaderboard seems easily cheated/gamed. once enough eyes are on it it will be worthless

bearjaws 3y ago
Kind of, ChatBot Arena was updated on Dec 7th, so hard to know for sure if Bard is just fine tuned against it. That being said Chatbot Arena is a pretty wide variety of test scenarios. If fine tuning made the model perfect, all the small models would get similar scores to GPT4, which they don't. Essentially it ranks how people believe a ChatBot should respond, rather than just zero shot, 1 shot and COT type benchmarks.
- deleted 3y ago
  [deleted]
karmasimida 3y ago
I don't think Google would specifically targeting this leaderboard just for marketing purpose though.
- kossTKR 3y ago
  Why not? Their entire presentation a month ago was fake.
  - karmasimida 3y ago
    Because this leaderboard has a very limited audience, and second only to GPT-4 turbo isn't good marketing regardless.
khimaros 3y ago
these scores are based on feedback from users who write any prompt they like and then blindly choose the best response. nothing to game.
deleted 3y ago
[deleted]
modeless 3y ago
How would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.
- pptr 3y ago
  I think there is a theoretical threat for models hosted on an external server. You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated. Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc. I doubt any of the big players are doing that, but you never know.
- visarga 3y ago
  I could tell from the second interaction which was Bard Pro because it uses a specific formatting with 2 levels of indentation I never saw in other models.

Kind of, ChatBot Arena was updated on Dec 7th, so hard to know for sure if Bard is just fine tuned against it. That being said Chatbot Arena is a pretty wide variety of test scenarios. If fine tuning made the model perfect, all the small models would get similar scores to GPT4, which they don't. Essentially it ranks how people believe a ChatBot should respond, rather than just zero shot, 1 shot and COT type benchmarks.

deleted 3y ago
[deleted]

karmasimida 3y ago

I don't think Google would specifically targeting this leaderboard just for marketing purpose though.

kossTKR 3y ago
Why not? Their entire presentation a month ago was fake.
- karmasimida 3y ago
  Because this leaderboard has a very limited audience, and second only to GPT-4 turbo isn't good marketing regardless.

khimaros 3y ago

these scores are based on feedback from users who write any prompt they like and then blindly choose the best response. nothing to game.

deleted 3y ago

[deleted]

modeless 3y ago

How would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.

pptr 3y ago
I think there is a theoretical threat for models hosted on an external server. You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated. Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc. I doubt any of the big players are doing that, but you never know.
visarga 3y ago
I could tell from the second interaction which was Bard Pro because it uses a specific formatting with 2 levels of indentation I never saw in other models.