It's because "runaway AI" implies a degree of superhuman skill at tasks including, at least, AI programming. (The reasoning goes that if it can reprogram itself

It's because "runaway AI" implies a degree of superhuman skill at tasks including, at least, AI programming. (The reasoning goes that if it can reprogram itself to improve its core intelligence beyond what it was initially programmed with, it must itself be pretty capable as an AI programmer, possibly better than the best programmers. If it can't, then it's not really dangerously smart). But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off. Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented. We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.