4 ms·

This future sounds like the present to me. The underlying worry in this article is that intelligence begets more intelligence and that the new forms of intelli

by dwiel 15y ago

This future sounds like the present to me. The underlying worry in this article is that intelligence begets more intelligence and that the new forms of intelligence might not have the same morals/goals as the author. Beneath this worry is that due to its exponential growth, intelligence and thus power quickly concentrates. Any thing/life that is not directly supporting whatever goals this new intelligence has is at great existential risk. We hope that these goals include benevolence towards other, less powerful forms of life. This all sounds reasonable, however they are held back by the idea that AI must look like a really smart human that is remotely controlling a robot/computer interface. The reality is that there is a lot of intelligence which looks nothing like us. The 'intelligence' of a market would be one basic example. If we take this more abstract/literal view of Artificially/human created intelligence, then most of what the article worries about is already upon us. Some people have so much wealth that we might as well consider them augmented. In many cases their interface to their augmented intelligence is another human, or many humans but that does not take away from their power. they have hoards of wealth at their disposal wealth is power and can currently be exchanged for both human and computer intelligence that purchased intelligence is then used to further increase the wealth/power/intelligence of the agent (billionaire/corporation) NOTE: The richest 0.000000044% of the world's population have the same amount of wealth as the bottom 8.8%. http://www.stwr.org/poverty-inequality/key-facts.html The result is that the rich/powerful/intelligent get more so, and everyone else must hope that they can somehow serve the few or that the few will be benevolent. If you'd rather avoid the rich/99% memes, try to describe our current market to someone from 1000 AD, and it sure sounds like an artificial intelligence: It grows. No single person controls it. It controls how resources and human labor are allocated in such a way so as to optimize its own growth. There are cases where people tried to take over the job of the market and they almost always fail (or always fail, depending on who you ask). Sure sounds like Artificially created intelligence to me. Either way, augmented billionaire, ultra-powerful cultural meme or human like intelligence in a machine, he seems to imply that our best bet is to teach and install morals and values in our 'children', with the hope that they will be benevolent. ###### I think that creating decentralization and diversity is more important than moral/value education; especially if you take the view that we are already in the midst of it all. One strong measure of the health of an ecological system is its diversity. This is partially because a monoculture is risky - it has a single point of failure. An ecosystem which is diverse is also an ecosystem which has the resources and stability to explore and learn. This is required if an ecosystem is going to survive change, as ours is now.

dncrane 15y ago
"This all sounds reasonable, however they are held back by the idea that AI must look like a really smart human that is remotely controlling a robot/computer interface." I think you misunderstood their position. The only assumption is that the AI must have some goal, and the point is that, if there's no term for humans in that goal, then we will likely be destroyed, not due to malice from any superintelligence, but indifference -- we are made out of resources that the AI could use to achieve its goals. I think you're vastly underestimating how much better than us at achieving goals a superintelligence would be. A superintelligence would be a lot more powerful than any augmented billionaire, in the same way that a human is more powerful than the head of any wolf pack. The worry isn't about slightly augmented humans or slightly superhuman intelligence. See these two papers for good arguments as to why human-level AI will likely result in an "intelligence explosion": Yudkowsky (2008), "AI as a Negative and Positive Factor in Global Risk" http://singinst.org/upload/artificial-intelligence-risk.pdf http://singinst.org/upload/artificial-intelligence-risk.pdf Chalmers (2010), The Singularity: A Philosophical Analysis http://consc.net/papers/singularityjcs.pdf http://consc.net/papers/singularityjcs.pdf
- dwiel 15y ago
  Thanks for the comment and links. I had only ever read 'science fiction' like Ian Banks and Charles Stross Accelerando, but never any of the 'non-fiction'. The linked articles were interesting. I understand that the risk of an AI that wants the mass of the entire solar system as its own and quickly becomes a matrioska brain. However, I question the idea that we aren't already there. I would argue that an augmented billionaire or better yet, a market economy compares to someone in the poorest 5% of the world just like a modern human compares to a wolf pack. A billionaire can decide not just to go to space, but to build an industry out of it. The poor can't find enough food. That is a huge difference. Is there any research into quantifying these types of differences? Sure a human level computer AI gets 'free' speed doubling every 18 months, but so does the intelligence that surrounds an augmented human. Just look at how much more intelligence we have available to us now as compared to 20 years ago. I agree that there will be/is an intelligence explosion. My point is simply that we are already in the midst of it, or that there is no single instant to point at and say 'that is when the singularity started.' I think that this is an important point to make because it changes the framing of the question from "How can we survive abstract superintelligence explosion of the unknown future" to "How can we survive the existing intelligence explosion." From "How can we teach the superintelligence we are bound to create to be nice" to "How can we convince the existing superintelligence to be nice" and/or "what changes in our social/governmental/memetic structure should we change to survive the explosion we are experiencing?" Also, if we view ourselves as already being in the intelligence explosion we can look at how existing superintelligences treat other less intelligent beings to see where our culture is likely to head as the explosion continues. If we don't like how superintelligences treat lessintelligences now, then maybe we should figure out why and how to change it. The framing that the article provides sounds about as silly as a pack of wolves discussing tactics they will use to make sure their new human creations focus all of their energy on catching rabbits; so I tried to come at it from an angle that has a hint of pragmatism and practicality.

by dwiel 15y ago

dncrane 15y ago

"This all sounds reasonable, however they are held back by the idea that AI must look like a really smart human that is remotely controlling a robot/computer interface." I think you misunderstood their position. The only assumption is that the AI must have some goal, and the point is that, if there's no term for humans in that goal, then we will likely be destroyed, not due to malice from any superintelligence, but indifference -- we are made out of resources that the AI could use to achieve its goals. I think you're vastly underestimating how much better than us at achieving goals a superintelligence would be. A superintelligence would be a lot more powerful than any augmented billionaire, in the same way that a human is more powerful than the head of any wolf pack. The worry isn't about slightly augmented humans or slightly superhuman intelligence. See these two papers for good arguments as to why human-level AI will likely result in an "intelligence explosion": Yudkowsky (2008), "AI as a Negative and Positive Factor in Global Risk" http://singinst.org/upload/artificial-intelligence-risk.pdf http://singinst.org/upload/artificial-intelligence-risk.pdf Chalmers (2010), The Singularity: A Philosophical Analysis http://consc.net/papers/singularityjcs.pdf http://consc.net/papers/singularityjcs.pdf

dwiel 15y ago
Thanks for the comment and links. I had only ever read 'science fiction' like Ian Banks and Charles Stross Accelerando, but never any of the 'non-fiction'. The linked articles were interesting. I understand that the risk of an AI that wants the mass of the entire solar system as its own and quickly becomes a matrioska brain. However, I question the idea that we aren't already there. I would argue that an augmented billionaire or better yet, a market economy compares to someone in the poorest 5% of the world just like a modern human compares to a wolf pack. A billionaire can decide not just to go to space, but to build an industry out of it. The poor can't find enough food. That is a huge difference. Is there any research into quantifying these types of differences? Sure a human level computer AI gets 'free' speed doubling every 18 months, but so does the intelligence that surrounds an augmented human. Just look at how much more intelligence we have available to us now as compared to 20 years ago. I agree that there will be/is an intelligence explosion. My point is simply that we are already in the midst of it, or that there is no single instant to point at and say 'that is when the singularity started.' I think that this is an important point to make because it changes the framing of the question from "How can we survive abstract superintelligence explosion of the unknown future" to "How can we survive the existing intelligence explosion." From "How can we teach the superintelligence we are bound to create to be nice" to "How can we convince the existing superintelligence to be nice" and/or "what changes in our social/governmental/memetic structure should we change to survive the explosion we are experiencing?" Also, if we view ourselves as already being in the intelligence explosion we can look at how existing superintelligences treat other less intelligent beings to see where our culture is likely to head as the explosion continues. If we don't like how superintelligences treat lessintelligences now, then maybe we should figure out why and how to change it. The framing that the article provides sounds about as silly as a pack of wolves discussing tactics they will use to make sure their new human creations focus all of their energy on catching rabbits; so I tried to come at it from an angle that has a hint of pragmatism and practicality.