5 ms·

> How can it not be obvious to you It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning. I feel like everything I've r

> How can it not be obvious to you It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning. I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing. Do you have any better resources that you could share?

hackinthebochs 3y ago
The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test. From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect. Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer. If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
patrec 3y ago
OK, which of the following propositions do you disagree with? 1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas. 2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying. 3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks. 4. AIs that are much smarter than any human pose even graver dangers. 5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage. 6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined. I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows. For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?
- revelio 3y ago
  Replying here, crossing over from the other thread. Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart". Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart. Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell. But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!" Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?

hackinthebochs 3y ago

The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test. From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect. Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer. If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.

patrec 3y ago

OK, which of the following propositions do you disagree with? 1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas. 2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying. 3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks. 4. AIs that are much smarter than any human pose even graver dangers. 5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage. 6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined. I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows. For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?

revelio 3y ago
Replying here, crossing over from the other thread. Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart". Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart. Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell. But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!" Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?