4 ms·
While I usually appreciate Venkatesh's perspectives, I think he's missing the point here. There is probably not "something it is like to be" the process of dif
by kragen 4y ago
While I usually appreciate Venkatesh's perspectives, I think he's missing the point here.
There is probably not "something it is like to be" the process of differential survival among pathogenic bacteria resulting from their genetic variation. Yet the resulting process is already capable of optimizing bacteria to survive in our wounds despite our antibiotics; by competing with us for access to our protein resources, it poses a serious threat to our survival, individually if not collectively.
If the optimization process involved were orders of magnitude faster, it could be a much bigger threat to our survival, at least if the metric it's optimizing trades off against our survival in some way. (Archaeans are subject to the same evolutionary optimization process as S. aureus, but because they're not adapted to the evolutionary niches in our bodies, antibiotic-resistant archaeans probably pose no threat.)
None of this reasoning depends on the SIILTBness or quality of experience or anthropomorphicity or mammalian-ness or "intention" or "sentience" of the optimization process in question. It's just about how much the optimization process's loss function trades off against human values, and whether it can outmaneuver humans and their institutions in real life, in the same way that AlphaGo can now outmaneuver us in the game of go, or corporations can outmaneuver individual humans in the economy, or governments can outmaneuver individual humans in warfare.
These last two examples clearly show that optimization processes whose loss functions are poorly aligned with human values can cause a lot of suffering, even when they are carried out by humans and even when the processes in question work very poorly in many ways. The humans can theoretically quit the game at any time ("What if they held a war and nobody came?") but in practice they face insuperable coordination problems in doing so. The fact that you can't fight City Hall doesn't imply that there is something it is like to be City Hall; the fact that you can't beat Boeing at winning DoD contracts doesn't mean there is something it is like to be Boeing.
We can reasonably expect that optimization processes that can easily outmaneuver human intelligence, operating in the world alongside it, will be much more dangerous to human values than bureaucracies, psychotic people with bricks, and microbial evolution.
That's the AI alignment problem. And the fear (that we will flub the AI alignment problem, disastrously) is something that Venkatesh's essay unfortunately failed to address at all.
There are other issues often brought up by fans of AI alignment which do depend on the question of what it is like to be a computer simulation: life extension through mind uploading, the ethics of causing suffering to simulated beings or terminating a simulation, teleportation, and so on. But, like the overlap with Effective Altruism, this is just a sociological coincidence, not a philosophical consequence; SIILTBness is totally irrelevant to AI alignment itself.
It's easy to get confused about this because the humans anthropomorphize optimization processes all the time, just like they anthropomorphize everything else. "The electron wants to go toward the positive charge," is very similar to, "AlphaGo wants to choose moves that improve its chances of winning at go," or, "The hypothetical paperclip maximizer is devoted to manufacturing an infinite number of paperclips."
It's easy to misread that "devoted to" as attributing awareness, creativity, passion, curiosity, and SIILTBness to the paperclip maximizer, and that misreading seems to be the mistake Venkatesh based his essay on; but it seems clear from the original context that that wasn't the intent. Precisely the opposite, in fact: http://extropians.weidai.com/extropians/0303/4140.html http://extropians.weidai.com/extropians/0303/4140.html
- joe_the_user 4y agoI think his argument is good in some ways and problematic in other ways. I think he distinguishes ordinary fears of AI from Hyperanthropomorphized fears. Humans training AI to, for example, kill in war, could result in mass death whether you have a human psychopath or a programming glitch involved. So he's not neglecting the bacteria analogy. Rather, he's addressing the problem of arguing from things like agency being ill-posed and their possibilities being incoherently understood. If you say "we don't know how agency is acquired so we should be worried about a random that seems like it has agency", you can wind-up with problem approach akin to "don't heat the compost pile or it might become superintelligence and kill us". However, I think this kind of argument is flawed because just because something like "general intelligence" isn't understood doesn't mean it doesn't exist. The definitions people make of it aren't good because it's not understood. I think a better way to approach this is that those who think AGI systems are a problem must create a more concrete, coherent idea of such system before their arguments can even be right or wrong. There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense.
- kragen 4y agoGeneral optimizing processes are qualitatively different from processes like the collapsing of bridges, the poisoning of ecosystems by chemicals, and the performance of Hellfire missile guidance systems on their way toward political dissidents; optimizing processes in bacterial and viral evolution already result in mass human death even though killing the humans isn't what those processes are optimizing for, and actually is a thing they "try" to reduce. (The bacteria or viruses trapped in the human corpse cannot reproduce and are usually quickly destroyed.) > There are simply too many problems with trying to reason about something that you accept you don't understand that effort framed that a destined to devolve into nonsense. Superficially this sounds smart (except for the part that says "understand that effort framed that a destined", which sounds like a GPT-2 glitch, but I assume you mean "understand that efforts to do so are destined") but it's profoundly wrong. I don't understand fluid dynamics, and weather is driven by fluid dynamics; nevertheless I can tell you that it will not rain today because it is a sunny day with a few cumulus clouds, and it never rains here on sunny days with a few cumulus clouds. (And even if my prediction were wrong, it wouldn't be nonsense, that is, "not even right or wrong"; a nonsense weather prediction sounds not like "it will not rain today" but rather like "it will triangle heavily last week" or "the sun will shine in the sky all night tonight.") Moreover, if we were to take seriously the idea that we can't reason about things we know we don't understand well enough for our arguments to have a truth-value, it would entail that we can't reason about not only weather but also people, materials, or orbital dynamics well enough for ideas like "the sun will shine tomorrow" to have a truth value. Similarly, "a superintelligent AI will kill us" is a prediction which is perfectly coherent within the usual parameters of weather predictions. We can disagree about whether AlphaZero counts as "superintelligent" or "AI", but it's clearly "more intelligent" than Deep Blue in the sense that it can learn to play novel games, and also IIRC it plays chess better than Deep Blue. We can argue about whether or not a particular situation qualifies as "dead", and about how many survivors are required to qualify as "us". We can argue about what time frame is relevant. Maybe Venkatesh would argue that a non-self-conscious optimization process wouldn't count as an "AI". But these semantic quibbles just make its truth-value a bit fuzzy; they don't eliminate it altogether, and neither do they eliminate our ability to reason about these possibilities.