> The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. That is really easy,

> The benchmark creation researchers have been doing their best to create difficult, lasting benchmarks that won't be broken by next week. That is really easy, say make it play pokemon as well as a 10 year old. That would take a very long time, I watched gemini play pokemon and its nowhere close to that even with all that help. The hard part isn't making a benchmark that wont be broken, its making a benchmark that is so easy that LLM can solve them but that is still hard for them to solve. Essentially what this means is that we have ran out of easy progress, and are now stumbling in the dark since we have no effective benchmarks to chase.