6 ms·
Superintelligence: fears, promises, and potentials
- deleted 11y ago[deleted]
- Moshe_Silnorin 11y agoI don't think Bostrom or Yudkowsky believe this: If one creates a human-level AGI with certain human-friendly goals, and allows it to self-modify freely, the odds are high that it will eventually self-modify into a condition where it no longer pursues the same goals it started out with I think their idea is a rational agent would have no incentive to change its terminal goals, as changing your utility function has to be of negative utility save for in some extreme edge cases that aren't likely to be relevant. The hard part is giving it the correct terminal goal. I also don't really think Bostrom is anthropomorphising. Basing your AI ideas on Piaget's theory of cognitive development, that seems like anthropomorphism - though likely entirely appropriate when the human mind is what you're attempting to mimic. Interesting article, brings back fond memories of Sl4,reading Ben, Yudkowsky, Robin Hanson, Gwern, and two Satoshi candidates argue was quite fun. I was convinced then that by 2015 nanotech or AI would have eaten the world. I'm a little more uncertain now, but we still have a couple days.
- joe_the_user 11y agoAI may be dangerous. AI can easily be dangerous without getting out of control of all humans. Rather, it could be very dangerous in the hands of a malevolent human - and we've not found a thing more inclined to harm humans and humanity than certain humans. That said, Yudkowsky's reasoning process about the situation seems fundamentally flawed - both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. IE, "rational" AI was discredited quite a while yet somehow the "rationalists" have wormed their way into the position of interpreters of future AI. Says more about human psychology than future prospects. This doesn't mean other disasters aren't lurking but I'd say it means that their particular disaster argument is untenable.
- Houshalter 11y ago>both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. These are just hypothetical scenarios they write about, to help explain problems with AI. I don't think they actually believe it will happen exactly like that. >However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. This is known as the Control Problem. The problem of controlling an AI's motivations so that they actually want to do what we tell them to.
- joe_the_user 11y agoAny truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. Computer programs don't care about anything inherently. Assuming people created programs with the ability to "know what we meant", the sensible thing would be to program them to care primarily about "doing what we mean". It seems like more or less an an anthropomorphization-driven illusion to believe that attaining "human intelligence" cause a machine to have all the unpredictable self-interesting-seek behaviors of humans. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.
- Houshalter 11y ago>the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program. But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all. How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press? >Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses. To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.
- mark_l_watson 11y agoA good overview of all sides of the debate. It is a long article but it is still a relatively quick way to get up to speed. Personally, I think the danger from AGIs is minimal compared to the uncertainty and dangers in this world - not now so much, but during the Cold War we faced the threat of extinction. The world is relatively safe now and I don't worry about AGIs. By the way, I think that the singularity and super intelligent AIs are in the distant future, even with the accelerating pace of innovation.
- jbpetersen 11y agoI think we're far more likely to see superintelligence emerge as a result of DAOs than as a result of AI.
- HillaryBriss 11y agoThe article lost me at "...it may well be best if advanced AI is developed in secret by a small elite group" How on earth could such a limitation be accomplished? It just doesn't seem plausible. If AI offers some organization somewhere in the world a competitive advantage -- how can development of that AI possibly be stopped?
- chubot 11y agoYeah it's not going to happen that way anyway. If you look at the history of science, multiple discovery is the rule rather than the exception. https://en.wikipedia.org/wiki/Multiple_discovery https://en.wikipedia.org/wiki/Multiple_discovery There are going to multiple groups... one of them could be small and elite I suppose. But we have no idea in general how it will play out. People seem to be scared of Google (OpenAI mentions them a lot), but the time frame is long enough that that's far from certain, and perhaps unlikely. I mean just like the Human Genome Project and Celera were within spitting distance of each other. I think that's how it will play out with AI.
- dbenhur 11y agoThe article argues against that proposition.
- haxel 11y agoWhat about the political implications of AGI? This concerns me most. We need to carefully consider who exactly controls such a powerful tool and what they use it for. Right now AGI is capital-intensive and risky: you need large-scale computing, top PhDs, and plenty of time. This means that only the most well-heeled organizations can justify earnest AGI development: big tech, government agencies, etc. Those orgs already have plenty of political power, and it seems to me that achieving AGI (or steps toward it) would amplify that power. Perhaps to the point of being unchallengeable. Do we want a political landscape where power is even more concentrated than now?
- btown 11y agoThose same characteristics of capital-intensity, talent-intensity, and risk are what prevent terrorist organizations from developing nuclear weapons. The open source, deconcentrated power paradigm works for most software. But it's very possible that AGI crosses the boundary to the point where we should think of non-proliferation as being worthwhile, despite the extent to which that entrenches power structures.
- argonaut 11y agoRight now, AGI does not exist.
- haxel 11y agoNot that we know of. If it does ever exist, how long will it take for us to learn of it? Powerful tools are often kept secret. And then, how long until the average person can leverage it themselves? The most powerful tools tend not to trickle down, if it can be helped.
- argonaut 11y agoEverything you've said is just speculation. Super intelligent AI won't appear out of nowhere. There will be semi-intelligent and normal-intelligent AI first.
- musha68k 11y agoThe best "fail-safe" for such an intelligence IMHO: It should be able to act out of love and yes, naturally we'd have to earn it if we want to be treated humanely as well. Love & understanding for all fellow animals, especially the weak and vulnerable. A whole new twist to the good old golden rule!
- Udik 11y agoI think this is an excellent and thorough critique of Bostrom's Superintelligence, which I've found to be a frustrating and sometimes infuriating read. My impression of it is that the abstruseness of the language conceals very few basic ideas, often repeated and poorly organized; vague statements and a great amount of unexplored or underexplored possibilities. I am actually very surprised by all the praise it has received. I'll probably pick it up again and try to parse it a second time more carefully, but the first impression was definitely that of an emperor with no clothes.