> the machine's intelligence exceeding ours strongly implies (to me, anyway) our inability to predict its behaviour Cognition and affect/motivation get conflat

> the machine's intelligence exceeding ours strongly implies (to me, anyway) our inability to predict its behaviour Cognition and affect/motivation get conflated a lot in AI discussions. Motivations don't magically arise from intelligence. They are not just acquired in upbringing either. Affect is a separate and innate system that requires its own explicit hardwiring. There is some imprinting, conditioning and other fine tuning with experiential feedback, but the basic drives and reward rules (and morals) have to be there from the beginning. Otherwise the person, artificial or not, will be catatonic.