2 ms·
There are enough such individuals that "unlucky string" multiplied by "observation count" is meaningful. My pet theory is just that publicly stated facts [0] ar
by hansvm 3y ago
There are enough such individuals that "unlucky string" multiplied by "observation count" is meaningful. My pet theory is just that publicly stated facts [0] are true and that users don't use the platform the same way.
[0a] The publicly stated facts being that OpenAI does more work behind the scenes than just ask a single model for probabilistic completions, and that they reduce the number of models being asked questions when under load.
[0b] The differences in platform usage being as simple as different questions (more or less susceptible to the changes) or different time-of-day or time-of-hour or other metric related to peak load.
It'd be fun for somebody to actually ask the same battery of questions over time and monitor the result distribution. Do you know of any such projects?
- visarga 3y agoChatbot Arena - a crowdsourced, randomized battle platform using 130K+ user votes to compute Elo ratings. This is a bit more reliable than the usual benchmarks, but slightly out of date because it needs to accumulate "battles" to rank new models. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- Traubenfuchs 3y agoLooks very gameable. > if(isBenchmarkInput(input)) ...