I really liked this. I think the author does a good job of steelmanning the AI alignment argument, and providing interesting food for thought on the subject.

I really liked this. I think the author does a good job of steelmanning the AI alignment argument, and providing interesting food for thought on the subject. Here's my responses to the arguments (since you didn't ask, and who cares what I think): Argument from Wooly Definitions: This isn't really an argument against, but just a "Maybe it's not a problem.". I think both Bostrom and Yudkowsky have said it's totally possible it's not a problem. The question is what probability you assign to "maybe intelligence can't be maximized to a problematic degree". There's not a lot of reason to assign a huge probability to that scenario. Even speeding up a 1-1 copy of a human brain to computer-speeds instead of synapse-speeds gets us into scary territory. Argument from Stephen Hawking's Cat: This one is more persuasive. It's essentially saying maybe the gap between human intelligence and "can trivially simulate human intelligence" is a really huge gap. The question kind of hinges on whether recursive self-improvement peters out at some point with the AI in the "Human to Cat" IQ gap, or whether it peters out somewhere in the "Human to Nematode" IQ gap, where we can almost simulate their brains entirely, and can certainly understand their motivations well enough to manipulate them. Again, we have a question of likelihood, and then we have to do the expected utility calculation (i.e. your estimated likelihood of the Human-Cat gap being the result has to be very small to offset the negative utility of complete annihilation). Argument from Einstein's Cat: This is essentially the same argument as above, but with force. The implication is that the cat is going to scratch the hell out of the human who forcibly tries to put it in a box. One element here I didn't address above is that the equivalent scenario isn't one human putting one cat who doesn't want to go into a box. The equivalent is one human trying to convince any cat anywhere by tricks, cajoling, petting, feeding into a box. That is, the AI just has to trick one human at some point into letting it onto the internet, etc. A human is totally capable of telling when it's going to get scratched and will know to avoid that cat and find another. The Argument from Emus: If you read the wiki page on the emu war, it looks like "only a few were killed" because of political pressure causing the army to quit after a few days of running into a little bit of trouble. Then the australian government instituted a bounty system, and wouldn't you know it, sufficiently motivated humans brought in 57,000 emus for bounties. This is an anecdote, not a strong argument, but it isn't a very reassuring anecdote. Maybe a few very determined humans would survive an AI onslaught? That doesn't seem like a conclusion I'd put into the "don't worry about it" bucket. The Argument From Slavic Pessimism: The author is arguing it will be very hard to align AI goals with human goals and we'll probably fuck it up even if we try really hard. I think everyone is in agreement on this one. But of course this is an argument for AI alarmism, not against it. Argument from Complex Motivations: I don't think the author seriously engaged with the orthogonality thesis, other than to just say "I don't believe it". Shruggy? The Argument from Actual AI: This boils down to arguing that the AI apocalypse isn't happening this year. I agree. Given how quickly very easy to use and flexible frameworks like PyTorch and Tensorflow emerged that allow even amateurs to implement bleeding edge techniques from the latest papers, I'm not super hopeful that "It's hard and our AI is bad" will continue to be the case for decades to come. The Argument from My Roommate: The author's roommate has a lot of competing evolutionary drives. Some of them say to conserve energy if there's no direct threat. Put another way: the paperclip maximizer might have a secondary goal of chilling out if it doesn't seem like any more paperclips are achievable at the moment. Still not a win for humans, just maybe the PM won't try venturing into space. Argument from Brain Surgery: It's pretty common to do brain surgery even on neural networks we have now. Train up a network on imagenet, rip off the top few layers, and retrain them for some new problem. Fundamentally, software and hardware designed by humans is much more understandable and decomposable than a human brain is (and we have no ethical qualms about doing crazy experiments on them, which hinders our ability to understand our own brains in vivo). It's true though that at present, deep neural networks operate in ways we don't understand and are hard to disentangle. Maybe that's fundamental to true intelligence, but probably not. The argument from Childhood: Understanding the real world requires spending real time, and that precludes hyper-explosive growth. This is true, and is a good reason to down-weight an intelligence explosion scenario. But we have good reason to think that it doesn't preclude it. There is a lot of work from OpenAI where a computer is trained up very quickly in simulation, then needs a very small amount of time in the real world to compensate for the differences between simulation and reality. The Argument from Gilligan's Island: It's a good point that humans' intelligence is dispersed, and that individually we aren't anywhere near as capable. AI has a particular advantage over us in this capacity: it can distribute its intelligence over multiple machines, but encounter non of the trust and incentive misalignments that humans must contend with when cooperating. I'd put this squarely in the "+1 for AI alarmists" bucket: we're handicapped in a way machines trivially aren't. It will be that much harder for us if an AI is misaligned. Outside arguments: All of these boil down to pattern matching. "Only nerds worry about this stuff. People who believe in this are megalomaniacs who place too much importance on themselves..." etc etc. These are weak arguments, and there are just as many weak anecdotal counterexamples where a person was worrying about something weird, and they turned out to be right. That weird person's name? Einstein. Overall impression: If I aggregate the strongest points from this talk, I'd probably phrase it something like: "Maybe there are diminishing returns to greater levels of intelligence, and humans are smart enough now that even exponentially more intelligent AIs will not be able to wipe us out completely." That's possible! We should probably at least spend some time thinking about what happens if that's not the case.