It is not good at that kind of novelty, but my impression is that the difficulty is that it is limited to a single pass through the network - such "loops" as th

It is not good at that kind of novelty, but my impression is that the difficulty is that it is limited to a single pass through the network - such "loops" as there are are "unrolled" within the network, into a very limited stack depth [0] (it would be very interesting to analyze these networks for self-similarity). If you want it to solve arbitrarily complex problems, you need to set up some sort of loop. People are already feeding the outputs back in as input in various primitive ways, but I suspect the real breakthrough will come when someone trains some sort of recursive transformer from scratch. (Assuming the current networks waste neurons in unrolling loops, we might possibly even see smaller models). [0] Try the following family of prompts: "_ is an example of _, which is an example of _, which is an example of _...." etc to a depth of your choosing. At some point it bottoms out and you can't get any more levels out of it.