6 ms·

Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson s

Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.

deegles 2y ago
'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng
- yowlingcat 2y ago
  This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
- brezelgoring 2y ago
  Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
  - thelittleone 2y ago
    I used to think I'd volunteer to go to Mars. But then I love the ocean, forests, fresh air, animals... and so on. So imagining myself in Mars' barren environment, missing Earth's nature feels downright terrible, which in turn, has taught me to take Earth's nature less for granted. Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
- bugbuddy 2y ago
  At the current Mars’ carrying capacity, one single person could be considered an overpopulation problem.
- newzisforsukas 2y ago
  https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learning-mandate-humans-not-just-machines/ https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learni... (2015) > What’s the most valid reason that we should be worried about destructive artificial intelligence? > I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
- kmacdough 2y ago
  Sentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions. Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
- in3d 2y ago
  Andrew Ng worked on facial recognition for a company with deep ties to the Chinese Communist Party. He’s the absolute worst person to quote.
  - bigcoke 2y ago
    omg no, the CCP!
- FranchuFranchu 2y ago
  Unfortunately, robots that kill people already exist. See: semi-autonomous war drones
moralestapia 2y ago
It would be akin to creating a "safe Dyson sphere", though; that's all it is. If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place? I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?" Congrats, Ilya. Eager to see what comes out of SSI.
TideAd 2y ago
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs. It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
- crazygringo 2y ago
  Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice. As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly. A physical on/off switch is a pretty powerful safeguard. (And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
  - hervature 2y ago
    I agree that an air-gapped AI presents little risk. Others will claim that it will fluctuate its internal voltage to generate EMI at capacitors which it will use to communicate via Bluetooth to the researcher's smart wallet which will upload itself to the cloud one byte at a time. People who fear AGI use a tautology to define AGI as that which we are not able to stop.
    - ben_w 2y ago
      I'm surprised to see a claim such as yours at this point. We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it. OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models. Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set. And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes. When it comes to humans stopping technology… well, when I was a kid, one pattern of joke was "I can't even stop my $household_gadget flashing 12:00": https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114 https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114
benreesman 2y ago
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like. Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
zild3d 2y ago
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
- benreesman 2y ago
  Nuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents. It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb. Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
- danielmarkbruce 2y ago
  Folks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand. Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
whimsicalism 2y ago
I think it’s clear we are at least at the remotely resembling intelligence stage… idk seems to me like lots of people in denial.
Sharlin 2y ago
You think we should try to create an unsafe Dyson Sphere first? I don't think that's how engineering works.

deegles 2y ago

'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng

yowlingcat 2y ago
This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
brezelgoring 2y ago
Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
- thelittleone 2y ago
  I used to think I'd volunteer to go to Mars. But then I love the ocean, forests, fresh air, animals... and so on. So imagining myself in Mars' barren environment, missing Earth's nature feels downright terrible, which in turn, has taught me to take Earth's nature less for granted. Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
bugbuddy 2y ago
At the current Mars’ carrying capacity, one single person could be considered an overpopulation problem.
newzisforsukas 2y ago
https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learning-mandate-humans-not-just-machines/ https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learni... (2015) > What’s the most valid reason that we should be worried about destructive artificial intelligence? > I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
kmacdough 2y ago
Sentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions. Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
in3d 2y ago
Andrew Ng worked on facial recognition for a company with deep ties to the Chinese Communist Party. He’s the absolute worst person to quote.
- bigcoke 2y ago
  omg no, the CCP!
FranchuFranchu 2y ago
Unfortunately, robots that kill people already exist. See: semi-autonomous war drones

moralestapia 2y ago

It would be akin to creating a "safe Dyson sphere", though; that's all it is. If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place? I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?" Congrats, Ilya. Eager to see what comes out of SSI.

TideAd 2y ago

So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs. It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.

crazygringo 2y ago
Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice. As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly. A physical on/off switch is a pretty powerful safeguard. (And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
- hervature 2y ago
  I agree that an air-gapped AI presents little risk. Others will claim that it will fluctuate its internal voltage to generate EMI at capacitors which it will use to communicate via Bluetooth to the researcher's smart wallet which will upload itself to the cloud one byte at a time. People who fear AGI use a tautology to define AGI as that which we are not able to stop.
  - ben_w 2y ago
    I'm surprised to see a claim such as yours at this point. We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it. OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models. Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set. And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes. When it comes to humans stopping technology… well, when I was a kid, one pattern of joke was "I can't even stop my $household_gadget flashing 12:00": https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114 https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114

benreesman 2y ago

InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like. Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.

zild3d 2y ago

The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors

benreesman 2y ago
Nuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents. It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb. Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
danielmarkbruce 2y ago
Folks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand. Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.

whimsicalism 2y ago

I think it’s clear we are at least at the remotely resembling intelligence stage… idk seems to me like lots of people in denial.

Sharlin 2y ago

You think we should try to create an unsafe Dyson Sphere first? I don't think that's how engineering works.