3 ms·
That's why the sample size is important, since a large sample size would offset a single bad actor. LMSys does have some anti-bot measures (e.g. Cloudflare) wh
by minimaxir 2y ago
That's why the sample size is important, since a large sample size would offset a single bad actor.
LMSys does have some anti-bot measures (e.g. Cloudflare) which makes it infeasible to manipulate the leaderboard for marketing.
- Tiberium 2y agoLMSys' Cloudflare protection is sadly trivial to bypass - just search for "LMSYS Chatbot Arena: Benchmarking LLMs in the Wild" (with quotes) in Censys and you'll find the origin host. It's a common misconfiguration for services that use Cloudflare as a proxy but then forget to actually make the origin server only respond to requests originating from CF IPs. Although I don't know if anyone's actually botting it.