This is a very cool result, although I don't think it implies that scale will create reasoning. Current AIs are fuzzy mad-libs engines. They probabilistically

This is a very cool result, although I don't think it implies that scale will create reasoning. Current AIs are fuzzy mad-libs engines. They probabilistically fill in the blank, based on the statistics of all the example text they've seen in training. In order to predict code successfully, most certainly there's longer-range multi-hop patterns between tokens. (A variable multiple lines away, a variable whose meaning/type/value changes after intermediate lines, ...) Regular english has far less of this. So it's not terribly surprising to me that including code in training is more effective, relative to not training on it at all. The question then is say we train on all text ever written in all languages. Including books, code, reddit comments, everything. Could the current architectures and training objectives produce a reasoning AGI? Or are we missing something? Frankly I think no one really knows for sure yet. I suspect we're missing something and "fill in the blank" is not the key to the universe.