I don't think trying to shove this model into a gradient-descent framework makes the most sense here. I'm an ML (industry) researcher and I highly doubt that AG

I don't think trying to shove this model into a gradient-descent framework makes the most sense here. I'm an ML (industry) researcher and I highly doubt that AGI will be achieved with gradient descent on neural networks alone. Those may play a small role somewhere in the stack but the orchestration and reasoning will be managed by something else entirely. Neural networks today are fancy MLE machines -- nowhere close to reasoning machines, which require an "understanding" (whatever that means in this context) of dynamics with respect to the environment. Seems more appropriate to start with a population of agents who reproduce at a rate proportional to their recent rewards, and allow them to die off at a rate inversely proportional to the same, a la a continuously-evolving genetic algorithm setup. You may have to modify the reward function to disincentivize behaviors which cause systemwide collapse, but that goes without saying.