3 ms·

I think his argument is good in some ways and problematic in other ways. I think he distinguishes ordinary fears of AI from Hyperanthropomorphized fears. Human

I think his argument is good in some ways and problematic in other ways. I think he distinguishes ordinary fears of AI from Hyperanthropomorphized fears. Humans training AI to, for example, kill in war, could result in mass death whether you have a human psychopath or a programming glitch involved. So he's not neglecting the bacteria analogy. Rather, he's addressing the problem of arguing from things like agency being ill-posed and their possibilities being incoherently understood. If you say "we don't know how agency is acquired so we should be worried about a random that seems like it has agency", you can wind-up with problem approach akin to "don't heat the compost pile or it might become superintelligence and kill us". However, I think this kind of argument is flawed because just because something like "general intelligence" isn't understood doesn't mean it doesn't exist. The definitions people make of it aren't good because it's not understood. I think a better way to approach this is that those who think AGI systems are a problem must create a more concrete, coherent idea of such system before their arguments can even be right or wrong. There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense.

kragen 4y ago
General optimizing processes are qualitatively different from processes like the collapsing of bridges, the poisoning of ecosystems by chemicals, and the performance of Hellfire missile guidance systems on their way toward political dissidents; optimizing processes in bacterial and viral evolution already result in mass human death even though killing the humans isn't what those processes are optimizing for, and actually is a thing they "try" to reduce. (The bacteria or viruses trapped in the human corpse cannot reproduce and are usually quickly destroyed.) > There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense. Superficially this sounds smart (except for the part that says "understand that effort framed that a destined", which sounds like a GPT-2 glitch, but I assume you mean "understand that efforts to do so are destined") but it's profoundly wrong. I don't understand fluid dynamics, and weather is driven by fluid dynamics; nevertheless I can tell you that it will not rain today because it is a sunny day with a few cumulus clouds, and it never rains here on sunny days with a few cumulus clouds. (And even if my prediction were wrong, it wouldn't be nonsense, that is, "not even right or wrong"; a nonsense weather prediction sounds not like "it will not rain today" but rather like "it will triangle heavily last week" or "the sun will shine in the sky all night tonight.") Moreover, if we were to take seriously the idea that we can't reason about things we know we don't understand well enough for our arguments to have a truth-value, it would entail that we can't reason about not only weather but also people, materials, or orbital dynamics well enough for ideas like "the sun will shine tomorrow" to have a truth value. Similarly, "a superintelligent AI will kill us" is a prediction which is perfectly coherent within the usual parameters of weather predictions. We can disagree about whether AlphaZero counts as "superintelligent" or "AI", but it's clearly "more intelligent" than Deep Blue in the sense that it can learn to play novel games, and also IIRC it plays chess better than Deep Blue. We can argue about whether or not a particular situation qualifies as "dead", and about how many survivors are required to qualify as "us". We can argue about what time frame is relevant. Maybe Venkatesh would argue that a non-self-conscious optimization process wouldn't count as an "AI". But these semantic quibbles just make its truth-value a bit fuzzy; they don't eliminate it altogether, and neither do they eliminate our ability to reason about these possibilities.

kragen 4y ago

General optimizing processes are qualitatively different from processes like the collapsing of bridges, the poisoning of ecosystems by chemicals, and the performance of Hellfire missile guidance systems on their way toward political dissidents; optimizing processes in bacterial and viral evolution already result in mass human death even though killing the humans isn't what those processes are optimizing for, and actually is a thing they "try" to reduce. (The bacteria or viruses trapped in the human corpse cannot reproduce and are usually quickly destroyed.) > There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense. Superficially this sounds smart (except for the part that says "understand that effort framed that a destined", which sounds like a GPT-2 glitch, but I assume you mean "understand that efforts to do so are destined") but it's profoundly wrong. I don't understand fluid dynamics, and weather is driven by fluid dynamics; nevertheless I can tell you that it will not rain today because it is a sunny day with a few cumulus clouds, and it never rains here on sunny days with a few cumulus clouds. (And even if my prediction were wrong, it wouldn't be nonsense, that is, "not even right or wrong"; a nonsense weather prediction sounds not like "it will not rain today" but rather like "it will triangle heavily last week" or "the sun will shine in the sky all night tonight.") Moreover, if we were to take seriously the idea that we can't reason about things we know we don't understand well enough for our arguments to have a truth-value, it would entail that we can't reason about not only weather but also people, materials, or orbital dynamics well enough for ideas like "the sun will shine tomorrow" to have a truth value. Similarly, "a superintelligent AI will kill us" is a prediction which is perfectly coherent within the usual parameters of weather predictions. We can disagree about whether AlphaZero counts as "superintelligent" or "AI", but it's clearly "more intelligent" than Deep Blue in the sense that it can learn to play novel games, and also IIRC it plays chess better than Deep Blue. We can argue about whether or not a particular situation qualifies as "dead", and about how many survivors are required to qualify as "us". We can argue about what time frame is relevant. Maybe Venkatesh would argue that a non-self-conscious optimization process wouldn't count as an "AI". But these semantic quibbles just make its truth-value a bit fuzzy; they don't eliminate it altogether, and neither do they eliminate our ability to reason about these possibilities.