We don't know how to do this yet, but humans have things which are a bit like utility functions, "built in" in a way we don't understand. However it's done, it

We don't know how to do this yet, but humans have things which are a bit like utility functions, "built in" in a way we don't understand. However it's done, it leads to self-improving behaviour in humans (many humans value learning and becoming better at things). Just because we don't know how to do it with deep learning and/or neural networks doesn't mean that it's a) not possible with those techniques, and b) that it's not possible with some other technique. (When a neural network becomes sophisticated enough to model its own workings and its relation to the environment, I don't see a knockdown argument for why it couldn't in principle optimise itself, given help from the outside.)