> I’m convinced we need to essentially create a simulation where something with agency can develop on its own in order to reach truly interesting AGI. That is

> I’m convinced we need to essentially create a simulation where something with agency can develop on its own in order to reach truly interesting AGI. That is fair, and some people do work in that direction. However, the step from "can develop on its own" to "will develop on its own" is highly non-trivial. And seeing the results from the really smart people that work in this direction, I am very much not inclined to work in that direction. There simply does not seem to be a good way to make progress. > Intelligence is, in comparison, an incredibly complex phenomenon that seems a lot harder to separate from the inner workings of the biology and evolutionary history. You might claim that now, as we don't understand intelligence yet. However, in the same way it was claimed in the 19th century that people would never be able to fly. What I meant to say is that things that seem hard in nature, might actually be a lot simpler in technology. I actually don't believe that intelligence is an incredibly complex phenomenon. On the contrary, I reckon we are really close to cracking it. I put it on the same level as moon-bases and nuclear fusion, all reasonably likely to happen in the next 20 years.