The metaphor that might describe this paper is "iteration". I'd hazard to predict that we’ll likely see more iterations of the following loop in 2025: -> A new

The metaphor that might describe this paper is "iteration". I'd hazard to predict that we’ll likely see more iterations of the following loop in 2025: -> A new benchmark emerges with a novel evaluation method. -> A new model saturates the benchmark by acquiring the novel “behavior.” -> A new benchmark introduces yet another layer of novelty. -> Models initially fail until a lab discovers how to acquire the new behavior. Case in point: OpenAI tackled this last step by introducing a paradigm called deliberative alignment to tackle some of the ARC benchmarks. [1] Alongside all this technical iteration, there’s a parallel cycle of product iteration, aiming to generate $ by selling intelligent software. The trillion $ questions are around finding the right iterations on both technical and product dimensions. [1] https://openai.com/index/deliberative-alignment/ https://openai.com/index/deliberative-alignment/