Just my opinion, but AI should be controllable and should probably start by, as the "Bayesian brain" theorists say, developing a robust and reliable model of it

Just my opinion, but AI should be controllable and should probably start by, as the "Bayesian brain" theorists say, developing a robust and reliable model of its environment. The Bayesian brain aspect of this is the belief that what AI needs to do is to minimize its own prediction errors. I take that to mean that an AI should learn to be safe and predictable on human terms before it is allowed to start to diverge from human expectations. Even that sounds a bit scary.