3 ms·
General optimizing processes are qualitatively different from processes like the collapsing of bridges, the poisoning of ecosystems by chemicals, and the perfor
by kragen 4y ago
General optimizing processes are qualitatively different from processes like the collapsing of bridges, the poisoning of ecosystems by chemicals, and the performance of Hellfire missile guidance systems on their way toward political dissidents; optimizing processes in bacterial and viral evolution already result in mass human death even though killing the humans isn't what those processes are optimizing for, and actually is a thing they "try" to reduce. (The bacteria or viruses trapped in the human corpse cannot reproduce and are usually quickly destroyed.)
> There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense.
Superficially this sounds smart (except for the part that says "understand that effort framed that a destined", which sounds like a GPT-2 glitch, but I assume you mean "understand that efforts to do so are destined") but it's profoundly wrong. I don't understand fluid dynamics, and weather is driven by fluid dynamics; nevertheless I can tell you that it will not rain today because it is a sunny day with a few cumulus clouds, and it never rains here on sunny days with a few cumulus clouds. (And even if my prediction were wrong, it wouldn't be nonsense, that is, "not even right or wrong"; a nonsense weather prediction sounds not like "it will not rain today" but rather like "it will triangle heavily last week" or "the sun will shine in the sky all night tonight.")
Moreover, if we were to take seriously the idea that we can't reason about things we know we don't understand well enough for our arguments to have a truth-value, it would entail that we can't reason about not only weather but also people, materials, or orbital dynamics well enough for ideas like "the sun will shine tomorrow" to have a truth value.
Similarly, "a superintelligent AI will kill us" is a prediction which is perfectly coherent within the usual parameters of weather predictions. We can disagree about whether AlphaZero counts as "superintelligent" or "AI", but it's clearly "more intelligent" than Deep Blue in the sense that it can learn to play novel games, and also IIRC it plays chess better than Deep Blue. We can argue about whether or not a particular situation qualifies as "dead", and about how many survivors are required to qualify as "us". We can argue about what time frame is relevant. Maybe Venkatesh would argue that a non-self-conscious optimization process wouldn't count as an "AI". But these semantic quibbles just make its truth-value a bit fuzzy; they don't eliminate it altogether, and neither do they eliminate our ability to reason about these possibilities.