7 ms·

This is why I doubt if efforts like OpenAI will ultimately turn out to be beneficial or catastrophic. I understand their purpose of democratizing the power and

This is why I doubt if efforts like OpenAI will ultimately turn out to be beneficial or catastrophic. I understand their purpose of democratizing the power and benefits of AI technology broadly. However, if their platform can provide a critical component or developmental path to AGI in the future, the unrestrained distribution of the technology is similar to spreading the blueprint of nuclear warheads. It could result in an equivalent of nuclear arms race, but worse--because a small team of capable programmers of any rogue organizations could come up with new improvements. It is true that we are probably quite far from AGI, but nobody really knows how far. We could be closer than most experts think. Few experts in 2006 would expect deep neural networks to beat a Go champion or be able to describe visual scenes in natural language sentences within a decade. Unexpected things happen, and we are much further behind in terms of preventive technology like Friendly AI. (https://en.wikipedia.org/wiki/Friendly_artificial_intelligence https://en.wikipedia.org/wiki/Friendly_artificial_intelligen...) Given the existential risks involved, I would propose more careful handling of the technology. (I agree that opening up most other technologies is usually a positive thing.) See "Superintelligence" for comprehensive arguments against unfettered development of AGI: https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...

markdog12 10y ago
It's a little repetitive and slow in places, but I wish more people read that book. It outlines some big issues with AGI. Changed my outlook on AI for sure.
YeGoblynQueenne 10y ago
>> the unrestrained distribution of the technology is similar to spreading the blueprint of nuclear warheads. There's an episode of Pinky and the Brain where Brain invents a magnification ray and gigantises Pinky and himself - so they're now giant rats [1]. They walk about some human metropolis, looming larger than skyscrapers and terrorising everyone. To cut the story short, Pinky happens and eventually the gun shoots everything in the world. ... so Pinky and the Brain are now normal-sized mice again. Or, well, they're giant mice, but everything in the world is also giant, so they're small. I think of that episode during discussions of the dangers of AGI and how "democratizing" it will eliminate those dangers. The problem is that AGI, like nuclear weapons, is not an equalizing force only- it is also a very destructive force, and so there is the danger that any loon with access to it could blow us all to kingdom come (with nuclear weapons, for sure; with AGI, perhaps). On the other hand, when nuclear weapons, or AGI, giant rays etc are not democratized we have the situation we're in today: only a few entities have access to them and they basically have free (military) reign... and there is still the danger that some loon might get to a position of power were they can push a button and -blamo. Which is why people are really worried about this sort of thing. Because once certain discoveries are made, there's no going back and at the same time the road ahead is full of danger. [1] https://www.youtube.com/watch?v=-fPcqu5aMyw https://www.youtube.com/watch?v=-fPcqu5aMyw
- visarga 10y ago
  We already have 7 billion of these H(uman)GI's walking around the world, with guns and dangerous tech. If the AGI will be much smarter, we shouldn't fear it the same we fear a dictator with nukes.
  - tossedaway334 10y ago
    "Smarter" has no nessisary or consequent intersection with "Shares human values" or even "Values human existence" https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer One of the reasons AGI will be so dangerous is the near certainty it will have utterly alien values and intentions. It seems exceedingly unlikely that we will be able to program in what consitutes "human values". We can't even begin to remotely agree and explain them fully to fellow humans, let alone what is basicly an alien intelligence. It is slightly ironic that people bring up Assimov's laws of robotics in this context, because pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. If something is very much smarter than humans, we cannot possibly hope to constrain it via some kind of faustian bargin agreement. Because in the end, the devil is always smarter than you are. If an AI kills you from either malice or indifference, or just that you are composed of plenty of paperclip-capable raw materials, you are dead all the same.
    - webmaven 10y ago
      > pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. The most chilling of these is the "Robot Dreams" short story.
      - visarga 10y ago
        I still think the AGI will have more to fear from people than people from it. We're together on this little boat called earth, 7 billion of us with access to weapons and many of us utterly crazy and willing to do stupid things for stupid reasons (see religion). We don't need AGI to blow ourselves up, plenty capable of that using our wetware neural nets. Imagine being locked in a prison cell with so many dangerous HGIs (Human GI's). I think the first thing for AGI would be to make sure we can't kill ourselves and it at the same time, by teaching us to become less stupid.
2468ben 10y ago
I did some heavy research in various forms of machine learning and AI in grad school 10 years ago, and the more experiments and tools I created, the more I saw this "digital nuclear arms race" and didn't want to be part of it. We don't know how many other people aren't working on this because of moral/ethical reasons. Of course, 99.9% of the world could be wary of genetic engineering, but that remaining 0.01% is enough to pursue research, get VC investment and drag the rest of us into that uncertain future. To continue the nuclear analogy, the Manhattan project was probably the most impressive engineering program in the history of the world, but it was driven by survival in a World War. They didn't build and drop the atomic bombs for fun. You'd think that working on a limitless virtual brain should have similarly serious motivations, not just the examples of "how do I drive to X but also shop for Y?" or "is that a monkey?". I know there are much grander societal goals with A.I. and the world really could become a "better place", but please sell society those goals, not the usual first-world problems. We already have enough people trying to destroy the world without these extra tools.
- drvdevd 10y ago
  Perhaps the general availability of AGI is antithetical to the notion of information privacy in the 21st century. And not just for individuals either, but for governments as well. I can imagine that control will only be possible with very deep, widespread monitoring.
Retric 10y ago
Repeat after me, "AI is not Magic." A super intelligent AI does not change the laws of physics or even get to predict the weather two weeks from now. Sure, they could learn to be a good hacker but the worlds best hacker can't get into air gaped systems. More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.
- chadgeidel 10y ago
  I'll admit this is purely hypothetical, but given that we know many people act on stuff they read on the Internet in an uncritical fashion - we cannot dismiss AI because it "doesn't have access to air gapped systems". A human has access to that system - a superintelligent AI could convince that human to "push the buttons". I don't actually believe this will happen, but I'm not 100% confident it wouldn't. I don't know what a superintelligence would look like!
  - Retra 10y ago
    Then your problem is humans, not AIs. And "pushing a button" is a severe understatement of what an AI would need to have a human do to cause the implied level of damage.
    - deleted 10y ago
      [deleted]
- adjkant 10y ago
  There's a subtle argument here philosophically that I think is crucially tied to AI: determinism. If the world is deterministic, a sufficiently powered and advanced AI with enough data could actually predict the weather, not just for two weeks, but for years. This also enables many other things viewed as magic. This, of course, is incredibly far off. And I don't see a flaw in your point on flawless security. Though, I doubt we have many, if any, truly flawless security systems in existence. We just don't know their flaws, making them good enough for now. The world could get really interesting once AI's start programming and writing app security to protect itself from malicious AI's.
laxatives 10y ago
OpenAI doesn't attempt to solve problems that even remotely resemble what you are describing. Hardly anyone claims to be working on AGI because AGI isn't a well defined problem. Its something cranks and the media tout because its excellent clickbait, along the lines of the singularity. All the efforts of ML solve problems with a concrete input space and a concrete output space. This means they solve very discrete problems and are not generalizable, unless they have similar looking model inputs and the model has the same limited range of "moves" it can make.
- nopinsight 10y ago
  I understand the limitations of current ML systems. That is why I mentioned 'a critical component'. Given that humans may use pattern recognition methods broadly similar in principle to deep neural nets, the conjecture is not without merit.

markdog12 10y ago

It's a little repetitive and slow in places, but I wish more people read that book. It outlines some big issues with AGI. Changed my outlook on AI for sure.

YeGoblynQueenne 10y ago

>> the unrestrained distribution of the technology is similar to spreading the blueprint of nuclear warheads. There's an episode of Pinky and the Brain where Brain invents a magnification ray and gigantises Pinky and himself - so they're now giant rats [1]. They walk about some human metropolis, looming larger than skyscrapers and terrorising everyone. To cut the story short, Pinky happens and eventually the gun shoots everything in the world. ... so Pinky and the Brain are now normal-sized mice again. Or, well, they're giant mice, but everything in the world is also giant, so they're small. I think of that episode during discussions of the dangers of AGI and how "democratizing" it will eliminate those dangers. The problem is that AGI, like nuclear weapons, is not an equalizing force only- it is also a very destructive force, and so there is the danger that any loon with access to it could blow us all to kingdom come (with nuclear weapons, for sure; with AGI, perhaps). On the other hand, when nuclear weapons, or AGI, giant rays etc are not democratized we have the situation we're in today: only a few entities have access to them and they basically have free (military) reign... and there is still the danger that some loon might get to a position of power were they can push a button and -blamo. Which is why people are really worried about this sort of thing. Because once certain discoveries are made, there's no going back and at the same time the road ahead is full of danger. [1] https://www.youtube.com/watch?v=-fPcqu5aMyw https://www.youtube.com/watch?v=-fPcqu5aMyw

visarga 10y ago
We already have 7 billion of these H(uman)GI's walking around the world, with guns and dangerous tech. If the AGI will be much smarter, we shouldn't fear it the same we fear a dictator with nukes.
- tossedaway334 10y ago
  "Smarter" has no nessisary or consequent intersection with "Shares human values" or even "Values human existence" https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer One of the reasons AGI will be so dangerous is the near certainty it will have utterly alien values and intentions. It seems exceedingly unlikely that we will be able to program in what consitutes "human values". We can't even begin to remotely agree and explain them fully to fellow humans, let alone what is basicly an alien intelligence. It is slightly ironic that people bring up Assimov's laws of robotics in this context, because pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. If something is very much smarter than humans, we cannot possibly hope to constrain it via some kind of faustian bargin agreement. Because in the end, the devil is always smarter than you are. If an AI kills you from either malice or indifference, or just that you are composed of plenty of paperclip-capable raw materials, you are dead all the same.
  - webmaven 10y ago
    > pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI. The most chilling of these is the "Robot Dreams" short story.
    - visarga 10y ago
      I still think the AGI will have more to fear from people than people from it. We're together on this little boat called earth, 7 billion of us with access to weapons and many of us utterly crazy and willing to do stupid things for stupid reasons (see religion). We don't need AGI to blow ourselves up, plenty capable of that using our wetware neural nets. Imagine being locked in a prison cell with so many dangerous HGIs (Human GI's). I think the first thing for AGI would be to make sure we can't kill ourselves and it at the same time, by teaching us to become less stupid.

2468ben 10y ago

I did some heavy research in various forms of machine learning and AI in grad school 10 years ago, and the more experiments and tools I created, the more I saw this "digital nuclear arms race" and didn't want to be part of it. We don't know how many other people aren't working on this because of moral/ethical reasons. Of course, 99.9% of the world could be wary of genetic engineering, but that remaining 0.01% is enough to pursue research, get VC investment and drag the rest of us into that uncertain future. To continue the nuclear analogy, the Manhattan project was probably the most impressive engineering program in the history of the world, but it was driven by survival in a World War. They didn't build and drop the atomic bombs for fun. You'd think that working on a limitless virtual brain should have similarly serious motivations, not just the examples of "how do I drive to X but also shop for Y?" or "is that a monkey?". I know there are much grander societal goals with A.I. and the world really could become a "better place", but please sell society those goals, not the usual first-world problems. We already have enough people trying to destroy the world without these extra tools.

drvdevd 10y ago
Perhaps the general availability of AGI is antithetical to the notion of information privacy in the 21st century. And not just for individuals either, but for governments as well. I can imagine that control will only be possible with very deep, widespread monitoring.

Retric 10y ago

Repeat after me, "AI is not Magic." A super intelligent AI does not change the laws of physics or even get to predict the weather two weeks from now. Sure, they could learn to be a good hacker but the worlds best hacker can't get into air gaped systems. More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.

chadgeidel 10y ago
I'll admit this is purely hypothetical, but given that we know many people act on stuff they read on the Internet in an uncritical fashion - we cannot dismiss AI because it "doesn't have access to air gapped systems". A human has access to that system - a superintelligent AI could convince that human to "push the buttons". I don't actually believe this will happen, but I'm not 100% confident it wouldn't. I don't know what a superintelligence would look like!
- Retra 10y ago
  Then your problem is humans, not AIs. And "pushing a button" is a severe understatement of what an AI would need to have a human do to cause the implied level of damage.
  - deleted 10y ago
    [deleted]
adjkant 10y ago
There's a subtle argument here philosophically that I think is crucially tied to AI: determinism. If the world is deterministic, a sufficiently powered and advanced AI with enough data could actually predict the weather, not just for two weeks, but for years. This also enables many other things viewed as magic. This, of course, is incredibly far off. And I don't see a flaw in your point on flawless security. Though, I doubt we have many, if any, truly flawless security systems in existence. We just don't know their flaws, making them good enough for now. The world could get really interesting once AI's start programming and writing app security to protect itself from malicious AI's.

laxatives 10y ago

OpenAI doesn't attempt to solve problems that even remotely resemble what you are describing. Hardly anyone claims to be working on AGI because AGI isn't a well defined problem. Its something cranks and the media tout because its excellent clickbait, along the lines of the singularity. All the efforts of ML solve problems with a concrete input space and a concrete output space. This means they solve very discrete problems and are not generalizable, unless they have similar looking model inputs and the model has the same limited range of "moves" it can make.

nopinsight 10y ago
I understand the limitations of current ML systems. That is why I mentioned 'a critical component'. Given that humans may use pattern recognition methods broadly similar in principle to deep neural nets, the conjecture is not without merit.