4 ms·

Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might ge

Considering how much progress we've been making with so little understanding of what's happening internally in these AI systems, the concern is that we might get to C rather quickly without anyone ever having drawn out a clear line from A->B->C. Here's an FAQ describing the basic concern: https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintelligence-faq https://www.lesswrong.com/posts/LTtNXM9shNM9AC2mp/superintel...

jazzyjackson 3y ago
This is the most specific worst case scenario I've come across so I'll respond to it: > For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again. I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)
- SpicyLemonZest 3y ago
  I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
  - jazzyjackson 3y ago
    That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop. Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.
- mrguyorama 3y ago
  It's important to note that the lesswrong's view of AGI is that it sees everything, knows everything, can do everything, can make humans do anything, and has motives directly harmful to us. These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic. It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.
mcguire 3y ago
Replace the term "superintelligence" with "God" in that FAQ. Do you still buy the conclusions? "A God is a mind that is much more intelligent than any human." "Go programs had gone from “dumber than children” to “God” in eighteen years, and “from never won a professional game” to “Godlike play” in six months." "God has an advantage that a human fighting a pack of lions doesn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically." "A God might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with." "Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical God is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways." (Oh, god, there's my personal bugbear: "bigger brain size correlates with greater intelligence". You know what else correlates with bigger brain size? Hint: I'm 6'5"; how much smarter am I than someone 5'1"?)
- MrScruff 3y ago
  Why would replacing 'superintelligence' with 'God' be insightful? Are you suggesting human intelligence is unique and the upper limit for intelligence? I mean we can argue about the difficulty and timeline for achieving human level artificial intelligence, but it seems unlikely that it's impossible.
  - mcguire 3y ago
    No, I'm suggesting the entire hype over "superintelligence" is religious in nature. There is little evidence that the current AI systems are anywhere near human intelligence and none that "more than human intelligence" is possible. Further, the abilities attributed to the "superintelligence" are completely godlike---it's omniscient and omnipotent, right? Nothing it can't do, potentially? So, if you're not worried about the extinction risk of God, why should you be concerned with the risk of "superintelligence"?
    - MrScruff 3y ago

jazzyjackson 3y ago

This is the most specific worst case scenario I've come across so I'll respond to it: > For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence. Now the superintelligence controls every computer in the world, including the ones that target nuclear weapons. At this point it can force humans to bargain with it, and part of that bargain might be enough resources to establish its own industrial base, and then we’re in humans vs. lions territory again. I believe there's a rift between doomers and eye-rollers because this kind of leap sounds either hollywood-hacker-sci-fi or plausible-and-obvious. The notion that software can re-deploy an improved version of itself without human intervention is just outside the realm of possibility to me (or somehow blackmailing or persuading a human to act alongside it?? Is that AI anymore or is that just a schizophrenic who thinks the computer is talking to him?)

SpicyLemonZest 3y ago
I don't think it can be reasonably disputed that software re-deploying an improved version of itself is plausible and obvious. Automated commits and automated deployments are both fairly common in modern build systems.
- jazzyjackson 3y ago
  That software exists to automatically apply changes to code doesn't have much bearing on software being able to introspect and decide what changes to apply, but maybe I'm moving the goalposts. The infrastructure exists for programs to edit their own deployment, what I'm doubting is that the software will have any idea of what can be improved without a human-in-the-loop. Compiler optimizations are a counterexample, could be called automated self-improvement -- but it's to do the same thing with fewer resources, what I'm looking for is a reason to believe an optimization could go the other way, have more capabilities with the same resources.
mrguyorama 3y ago
It's important to note that the lesswrong's view of AGI is that it sees everything, knows everything, can do everything, can make humans do anything, and has motives directly harmful to us. These are all taken as a given because the entire concept is just old testament god but with glowy parts. This is an essential part of the dogma, which is why there's never any sort of justification for it. Super smart computer is just assumed to be magic. It's plausible and obvious to them because "a super-intelligence can make anyone do anything", can reprogram any computer to it's will, and can handwave away literally technical deficiencies.

mcguire 3y ago

Replace the term "superintelligence" with "God" in that FAQ. Do you still buy the conclusions? "A God is a mind that is much more intelligent than any human." "Go programs had gone from “dumber than children” to “God” in eighteen years, and “from never won a professional game” to “Godlike play” in six months." "God has an advantage that a human fighting a pack of lions doesn’t – the entire context of human civilization and technology, there for it to manipulate socially or technologically." "A God might be able to analyze human psychology deeply enough to understand the hopes and fears of everyone it negotiates with." "Humans instinctively balance thousands of different considerations in everything they do; so far this hypothetical God is only balancing three (least cancer, quickest results, highest probability). To a human, it would seem maniacally, even psychopathically, obsessed with cancer curing. If this were truly its goal structure, it would go wrong in almost comical ways." (Oh, god, there's my personal bugbear: "bigger brain size correlates with greater intelligence". You know what else correlates with bigger brain size? Hint: I'm 6'5"; how much smarter am I than someone 5'1"?)

MrScruff 3y ago
Why would replacing 'superintelligence' with 'God' be insightful? Are you suggesting human intelligence is unique and the upper limit for intelligence? I mean we can argue about the difficulty and timeline for achieving human level artificial intelligence, but it seems unlikely that it's impossible.
- mcguire 3y ago
  No, I'm suggesting the entire hype over "superintelligence" is religious in nature. There is little evidence that the current AI systems are anywhere near human intelligence and none that "more than human intelligence" is possible. Further, the abilities attributed to the "superintelligence" are completely godlike---it's omniscient and omnipotent, right? Nothing it can't do, potentially? So, if you're not worried about the extinction risk of God, why should you be concerned with the risk of "superintelligence"?
  - MrScruff 3y ago