2 ms·
Sure but also see the benchmark creation to benchmark breaking race. The benchmark creation researchers have been doing their best to create difficult, lasting
by creer 1y ago
Sure but also see the benchmark creation to benchmark breaking race. The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. They are not illiterate idiots. And yet the LLMs (or LLM-based systems really) have been consistently so far breaking the benchmarks in absurd time.
It's hard to say that nothing significant is going on.
Also the fine article's entire point is that these are not points on a continuum.
- Jensson 1y ago> The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. That is really easy, say make it play pokemon as well as a 10 year old. That would take a very long time, I watched gemini play pokemon and its nowhere close to that even with all that help. The hard part isn't making a benchmark that wont be broken, its making a benchmark that is so easy that LLM can solve them but that is still hard for them to solve. Essentially what this means is that we have ran out of easy progress, and are now stumbling in the dark since we have no effective benchmarks to chase.