3 ms·
Ignoring the obvious ai slop webpage. I don't really trust the benchmark. It seems either pretty saturated, or inconsistent just based on the results.
by gunalx 17d ago
Ignoring the obvious ai slop webpage. I don't really trust the benchmark. It seems either pretty saturated, or inconsistent just based on the results.
- XCSme 17d agoWhat seems inconsistent? The coverage is quite small, only 22 tests. It's more to compare the cost/speed/consistency between models, given the same tasks.
- gunalx 16d agoRight. I got the feeling of it being saturated because all the top 5 fully completed it.
- XCSme 16d agoYeah, it's hard to find a single simple task that all models fail on, in low context length conditions. Also because models now are actually not that good on knowing things (domain knowledge), as they rely more on web search on tool use. So if I added a question, about some obscure fact, probably the SOTA models would fail it, but in practice they would find it with web search enabled. Not sure how to handle that. This is also why Gemini is on top, it's good enough at coding and instructions following, while having by far best general and domain specific knowledge.