I think it goes beyond that. When we say animals learn and humans in particular generate new knowledge by exploration, it's not just system I versus system II t

I think it goes beyond that. When we say animals learn and humans in particular generate new knowledge by exploration, it's not just system I versus system II thinking. You might legitimately not be able to answer question just by thinking at all. You have to go out in the world and find the answer. It might be as simple and walking around a building to see what's on the other side or it might as complicated as spending a few thousand years learning to build particle accelerators and sensors that can infer what is happening at a subatomic level in these, but you can't just "generate" an answer with sheer brainpower. This is a lot older than Kahneman. It's the classical split between rationalists and empiricists. AI maximalists and x-risk death cult fanatics are effectively being rationalists. They don't believe a sufficiently intelligent entity will be bound by the need to do science. It can, by sheer force of thought, figure out the answer to any question, develop a perfect course of action to achieve any goal, as quickly as electricity can traverse its own internal circuits. It's interesting to see the constraints on how they can even study this in the paper, though. You can only ever ask a computing agent to do what it even can do. If you want it to explore, it needs a toy world that maybe digitally encodes a reasonable stripped-down representation of the real world, but without a body, sensors, and full autonomy, it can't ever truly explore the real real world.