3 ms·

One of these things is real and happening now. It isn't potential. The other is a hypothetical future scenario that has literally no A->B->C line drawn to it,

One of these things is real and happening now. It isn't potential. The other is a hypothetical future scenario that has literally no A->B->C line drawn to it, pushed by people whose ability to feed themselves depends on attention farming. So no. Let's not focus on the bullshit one, and let's focus on the one that is hurting people in the now.

circuit10 3y ago
But there is a clear line of reasoning? AI has to be made with some kind of goal. Most goals don’t line up with humanity’s interests, and the AI will pursue them to the extreme as it’s designed for optimise aggressively for it. AI is easier to improve than biology, so once it gets good enough it will exponentially improve itself, as that allows it to achieve its goal better. Most solutions to this problem don’t actually work, we’ve yet to find a good one. It seems extremely short sighted to never consider any future situations that aren’t currently happening but seem likely to: https://xkcd.com/2278/ https://xkcd.com/2278/ There is a good real world example of how optimising for a goal aggressively can make the world unliveable for some species, which is humans destroying the environment for our economic growth. But at least there we do care about it and are trying to change it, because, among other reasons, we need the environment to survive ourselves. An AI wouldn’t necessarily need that, so it would treat us more like we treat bacteria or something, only keeping us around if necessary or if we don’t get in the way (and we will get in the way) It might sound silly but there is some solid reasoning behind it
- eropple 3y ago
  Let me rephrase: a clear line of reasoning that isn't an Old Testament god with some Tron lines.
lumenwrites 3y ago
So your suggestion is that we ignore the existential risk until after it occurs? > literally no A->B->C line drawn to it Spend a couple of hours to educate yourself about the issue, and make a convincing counterargument please: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... https://astralcodexten.substack.com/p/why-i-am-not-as-much-o https://astralcodexten.substack.com/p/why-i-am-not-as-much-o... https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... https://www.youtube.com/@RobertMilesAI/videos https://www.youtube.com/@RobertMilesAI/videos
- jazzyjackson 3y ago
  posting links to yudkowsky is just not going to convince people who don't already see it his way. he doesn't even try to be persuasive.
justinpombrio 3y ago
Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might get to C rather quickly without anyone ever having drawn out a clear line from A->B->C. Here's an FAQ describing the basic concern: https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintel...
- jazzyjackson 3y ago
  This is the most specific worst case scenario I've come across so I'll respond to it: > For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again. I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)
  - SpicyLemonZest 3y ago
    I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
    - jazzyjackson 3y ago
      That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop. Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.

circuit10 3y ago

But there is a clear line of reasoning? AI has to be made with some kind of goal. Most goals don’t line up with humanity’s interests, and the AI will pursue them to the extreme as it’s designed for optimise aggressively for it. AI is easier to improve than biology, so once it gets good enough it will exponentially improve itself, as that allows it to achieve its goal better. Most solutions to this problem don’t actually work, we’ve yet to find a good one. It seems extremely short sighted to never consider any future situations that aren’t currently happening but seem likely to: https://xkcd.com/2278/ https://xkcd.com/2278/ There is a good real world example of how optimising for a goal aggressively can make the world unliveable for some species, which is humans destroying the environment for our economic growth. But at least there we do care about it and are trying to change it, because, among other reasons, we need the environment to survive ourselves. An AI wouldn’t necessarily need that, so it would treat us more like we treat bacteria or something, only keeping us around if necessary or if we don’t get in the way (and we will get in the way) It might sound silly but there is some solid reasoning behind it

eropple 3y ago
Let me rephrase: a clear line of reasoning that isn't an Old Testament god with some Tron lines.

lumenwrites 3y ago

So your suggestion is that we ignore the existential risk until after it occurs? > literally no A->B->C line drawn to it Spend a couple of hours to educate yourself about the issue, and make a convincing counterargument please: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... https://astralcodexten.substack.com/p/why-i-am-not-as-much-o https://astralcodexten.substack.com/p/why-i-am-not-as-much-o... https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... https://www.youtube.com/@RobertMilesAI/videos https://www.youtube.com/@RobertMilesAI/videos

jazzyjackson 3y ago
posting links to yudkowsky is just not going to convince people who don't already see it his way. he doesn't even try to be persuasive.

justinpombrio 3y ago

Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might get to C rather quickly without anyone ever having drawn out a clear line from A->B->C. Here's an FAQ describing the basic concern: https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintel...

jazzyjackson 3y ago
This is the most specific worst case scenario I've come across so I'll respond to it: > For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again. I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)
- SpicyLemonZest 3y ago
  I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
  - jazzyjackson 3y ago
    That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop. Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.