I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since

I made that assumption explicitly, actually. To make it worse, it seems plausible that these kinds of dangerous AIs will be the most simple ones to build, since they won't require a lot of goal engineering. Intelligence is a superset of feedback loops. It is concerned with cases in which feedback loops are not sufficient to optimize the agents objectives, and attempts to reach them using learning and prediction. The complexity in behavior comes from interacting with a complex environment and having many model parameters; not (necessarily) from the initial goals. (As an intuition pump, have a look at GoogleMind's atari reinforcement learner. The essential parts of the code fit onto a single page [1].) [1] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner