Qualitatively, the answer seems to be to train for novelty. Which happens to also be how humans accomplish much of the complex stuff. Eg, One interpretation of

Qualitatively, the answer seems to be to train for novelty. Which happens to also be how humans accomplish much of the complex stuff. Eg, One interpretation of art is we have 8 billion or so humans with a finely trained neural net for recognising stuff. The artist is looking for novel ways of triggering those nets, and if they find something inspired then it is art. A great artist generally isn't trying to reproduce something that is known, they're trying to explore novel areas of a medium. So the real trick to building the world model is coming up with a good novelty metric. Still hard, but easier than developing a reward function. That gets the part of the training done that establishes a world model, then I'd assume it is possible to train that model to do a task by rewarding specific outcomes that it already knows how to achieve.