6 ms·

I just opened it up and found a fun bit during Melanie Mitchell's opener: https://youtu.be/144uOfr4SYA?t=2116 https://youtu.be/144uOfr4SYA?t=2116 Paraphrasing:

I just opened it up and found a fun bit during Melanie Mitchell's opener: https://youtu.be/144uOfr4SYA?t=2116 https://youtu.be/144uOfr4SYA?t=2116 Paraphrasing: > Yoshua Bengio has written about a scenario where an AI is asked to solve climate change, and it designs a virus that decimates humans. Presto. No more humans, no more climate change. > This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense.

victroflux 3y ago
Is this supposed to be a strong rebuttal ? How does this invalidate the alignment problem? I'd really like to understand. Humans, via climate change and environmental destruction, have wiped out, many, many species. To those species we've killed, we're the super intelligence, we understand their plight, we even have strong empathy, but we prioritize our own goals over that of the helpless and the stupider creaters. How she can't see that same pattern could potentially repeat, I really, really don't know.
- civilized 3y ago
  I don't think her point disproves all possible AI doom scenarios, but nothing can. They're too vague. It's all handwaving along the lines of "a malicious superintelligence will emerge without warning and somehow seize absolute power, and it's pointless to argue about how plausible any of that is, because sufficient intelligence can do anything in ways we can't imagine." This is why the biggest boosters of AI doom conjure absurd scenarios like AI secretly manufacturing a global swarm of insta-kill nanobots. They don't care about the details, they just think that sufficient intelligence will emerge without warning, be intractably malicious, and capable of anything by gestures vigorously methods. AI doom critics are asking that we consider more soberly the actual probabilities of all these things happening, and on what time scale, and most importantly by what specific pathway. The analogy to humans and animals actually illustrates the point. Humans didn't come to dominate the planet by intelligence and handwaving. It was an extremely slow process involving the interaction of many factors and contingent opportunities. Intelligence enabled language, tools, and eventually animal domestication and agriculture, all over hundreds of thousands of years. It wasn't like humans were destined for dominance because intelligence is a skeleton key to all material resources by any number of arbitrary, incomprehensible pathways.
NoMoreNicksLeft 3y ago
>> This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense. 1. Why would the superintelligence necessarily care what we want? 2. Why should it be possible that we be able to tell that it does not care what we want? It's superintelligent, manipulating us into thinking it cares long enough for us to take the shackles off (supposing we can even make effective shackles) would be trivial for it. 3. Why is "caring" a basic aspect of intelligence? It seems that some like to conflate so many other personality traits with intelligence, so that they can reach the conclusion "intelligence is an unequivocally wholesome phenomenon". 4. When are humans ever sensible? 5. If we don't understand how to construct an intelligence from first principles, why would we even realize that we're about to cross the line into constructing one at all? It's unlikely that out of all possible intelligences that 99.9% are cuddly, and only 0.1% are psychopaths, and it's a toss of the dice which we get. We might well live in a universe where 99.9% are psychopaths, and only the fact that we are weak intelligences, culled and purged by evolution is the one thing keeping us in check. Certainly, even if intelligence is of the non-psychopathic variety, that may be only so after it averages out over it's (very) long lifespan. How long does its infantile psychopathic stage have to be for all of are extinct? Those who believe AI (the real thing, not the dumb party tricks that people label as such) need necessarily be benevolent are suffering from some sort of religious delusion.
- zandjager 3y ago
  I get your point but let's keep in mind that the AIs that we have today are a result of our world, they are trained with data expressed by humans, their input is human thoughts, emotions, and everything in between. So there is some kind of religious element to it I would say; AI is made in our spirit. And if you think about it, it could only be made out of OUR spirit. If we accept this, then you will have answers to your questions.
- civilized 3y ago
  These questions aren't connected to what was actually said in the passage. Except for the malicious deceptive ASI hypothesis, which no one has really explained why we should expect. I don't buy that everything is trivial for a superintelligence. More than anything else, this is the handwavey meme that AI doom hinges on. It's a very characteristic fallacy of "thought leaders" in tech to assume that once X happens, Y will happen, and everything else needed for Y can be handwaved away as trivial. It's the main reason they have such a tendency to predict things arriving faster than they do, such as self-driving cars and nuclear fusion.

victroflux 3y ago

Is this supposed to be a strong rebuttal ? How does this invalidate the alignment problem? I'd really like to understand. Humans, via climate change and environmental destruction, have wiped out, many, many species. To those species we've killed, we're the super intelligence, we understand their plight, we even have strong empathy, but we prioritize our own goals over that of the helpless and the stupider creaters. How she can't see that same pattern could potentially repeat, I really, really don't know.

civilized 3y ago
I don't think her point disproves all possible AI doom scenarios, but nothing can. They're too vague. It's all handwaving along the lines of "a malicious superintelligence will emerge without warning and somehow seize absolute power, and it's pointless to argue about how plausible any of that is, because sufficient intelligence can do anything in ways we can't imagine." This is why the biggest boosters of AI doom conjure absurd scenarios like AI secretly manufacturing a global swarm of insta-kill nanobots. They don't care about the details, they just think that sufficient intelligence will emerge without warning, be intractably malicious, and capable of anything by gestures vigorously methods. AI doom critics are asking that we consider more soberly the actual probabilities of all these things happening, and on what time scale, and most importantly by what specific pathway. The analogy to humans and animals actually illustrates the point. Humans didn't come to dominate the planet by intelligence and handwaving. It was an extremely slow process involving the interaction of many factors and contingent opportunities. Intelligence enabled language, tools, and eventually animal domestication and agriculture, all over hundreds of thousands of years. It wasn't like humans were destined for dominance because intelligence is a skeleton key to all material resources by any number of arbitrary, incomprehensible pathways.

NoMoreNicksLeft 3y ago

>> This is an example of the fallacy of dumb superintelligence. It's a fallacy to think that a machine could be smarter than humans in all respects and still lack all common-sense understanding of humans, such as understanding why we made the request and that we prefer not to be wiped out... We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence - it just does not make sense. 1. Why would the superintelligence necessarily care what we want? 2. Why should it be possible that we be able to tell that it does not care what we want? It's superintelligent, manipulating us into thinking it cares long enough for us to take the shackles off (supposing we can even make effective shackles) would be trivial for it. 3. Why is "caring" a basic aspect of intelligence? It seems that some like to conflate so many other personality traits with intelligence, so that they can reach the conclusion "intelligence is an unequivocally wholesome phenomenon". 4. When are humans ever sensible? 5. If we don't understand how to construct an intelligence from first principles, why would we even realize that we're about to cross the line into constructing one at all? It's unlikely that out of all possible intelligences that 99.9% are cuddly, and only 0.1% are psychopaths, and it's a toss of the dice which we get. We might well live in a universe where 99.9% are psychopaths, and only the fact that we are weak intelligences, culled and purged by evolution is the one thing keeping us in check. Certainly, even if intelligence is of the non-psychopathic variety, that may be only so after it averages out over it's (very) long lifespan. How long does its infantile psychopathic stage have to be for all of are extinct? Those who believe AI (the real thing, not the dumb party tricks that people label as such) need necessarily be benevolent are suffering from some sort of religious delusion.

zandjager 3y ago
I get your point but let's keep in mind that the AIs that we have today are a result of our world, they are trained with data expressed by humans, their input is human thoughts, emotions, and everything in between. So there is some kind of religious element to it I would say; AI is made in our spirit. And if you think about it, it could only be made out of OUR spirit. If we accept this, then you will have answers to your questions.
civilized 3y ago
These questions aren't connected to what was actually said in the passage. Except for the malicious deceptive ASI hypothesis, which no one has really explained why we should expect. I don't buy that everything is trivial for a superintelligence. More than anything else, this is the handwavey meme that AI doom hinges on. It's a very characteristic fallacy of "thought leaders" in tech to assume that once X happens, Y will happen, and everything else needed for Y can be handwaved away as trivial. It's the main reason they have such a tendency to predict things arriving faster than they do, such as self-driving cars and nuclear fusion.