My favorite part of this scary tale is the assumption that a general AI would be highly motivated. Largely by definition, we can't stop a superintelligent a

My favorite part of this scary tale is the assumption that a general AI would be highly motivated. Largely by definition, we can't stop a superintelligent agent from doing what it wants to do. We could build 9 strong agents in a row that just gazed at their navels and proved graph theorems all day, but if the 10th one managed to turn the "motivation" dial all the way up during hyperparameter tuning, then it'll wipe us out in its quest to build more computers so it can prove more theorems.