You wouldn't do such a thing because you have a bunch of hard-coded goals provided by evolution, such as "don't destroy your own social status". We're not build

You wouldn't do such a thing because you have a bunch of hard-coded goals provided by evolution, such as "don't destroy your own social status". We're not building AIs by evolving them, and even if we did, we couldn't provide it with the same environment we evolved in, so there's no reason it would gain the same hard-coded goals. Why would an AGI even have the concept of goals being "stupid"? We've already seen simple AIs achieving goals by "stupid" means, e.g. playing the longest game of Tetris by leaving in on pause indefinitely. AGI is dangerous not because of potential misunderstanding, but because of potential understanding. The great risk is that it will understand its goal perfectly, and actually carry it out.