11 ms·

The Thinkers Behind Musk’s Fear: Russell, Bostrom, and Omohundro

77 points by Moshe_Silnorin 12y ago

joe_the_user 12y ago
It seems like AI has become more capable but has stayed in the realm of "dumb AI" (not incapable just not human-like). Indeed, by no longer aiming at duplicating human consciousness, machine learning has found more and more niches. But this doesn't mean that right now, the corporations and governments that are deploying this AI and indeed displacing human choice are "provably aligned with human values". Indeed, it almost sound like there's this hope that humans will build machines that treat people right after humans themselves have failed terribly in treating each right. I'm skeptical that this could work.
- Retra 12y ago
  You need to know _why_ humans don't treat each other right. Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well. Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.
  - landryraccoon 12y ago
    Human beings can't even agree on a definition of "treat each other right", let alone actually implement it. What makes you think we can program an AI to do it?
    - hedgew 12y ago
      We have pretty good ideas on how humans or other intelligent agents could "treat each other right". Philosophers, and people who study game theory in particular, have many answers to this problem. The prisoner's dilemma is a simple example of why treating everyone right is challenging.
    - Retra 12y ago
      1. Humans don't have to agree to something to program it. 2. What makes you think a computer can't do something just because you don't know how to do it? Maybe someone else does.
    - evanmoran 12y ago
      I think it's much easier to get 100 people to agree to high moral ideals then to try to get everyone at once. The reason this is possible is only the programmers have to agree.
    - eli_gottlieb 12y ago
      >Human beings can't even agree on a definition of "treat each other right", let alone actually implement it. If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy. Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical). Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics! Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
- dorfsmay 12y ago
  This is a post vs pre singularity problem. Post-singularity, the bots won't care about our rules of ethics, rules of engagement etc... Definitely something to worry about. Before we get there though, we have the issue of the dumb bots giving exponential powers to the few humans in charge. Today military and police personnel can refuse to execute an order that is wrong. Dumb bots won't.
  - joe_the_user 12y ago
    Today military and police personnel can refuse to execute an order that is wrong. Maybe, though we see more military and police adding in extra-behaviors that seem rather wrong. A dumb bot merely enforcing a given law might actually more desirable than present-day cops who apparently gratuitously murder civilians on a regular basis.
- ridgeguy 12y ago
  So long as (human, artificial) intelligences operate within an evolutionary dynamic, the definitions of what it means to "treat people right" (or even who's included in the class: people) will be embodied in the evolutionary winners. If it happens that human-friendly values degrade fitness in the AI arena, the winners won't have retained or evolved those values. If the values are evolution-neutral, well, perhaps. Evolution requires only variance, selection, and inheritance. I don't know how we'll develop useful ("strong") AI in a setting that excludes AI evolution. I think this means we can't guarantee a human-benign outcome for strong AI.
  - joe_the_user 12y ago
    Evolution is a theory of how biological systems change. Even in regard to biological systems, it's not a theory of what's better but simply what happens. However, outside biological systems, it's not really applicable at all. You can't say a rock, a broom or a computer evolve in the standard sense of Darwinian evolution - only biological systems have the tendency to reproduce themselves. Computers don't reproduce themselves, they are simply produced by us. If an AI is constructed, that will be what happens - human choice (or maybe chance) will be the agent, not biological evolution.
    - ridgeguy 12y ago
      I think evolution is more widely applicable than just to biological systems. To date, we've only studied its operation in biosystems, which, given our status as biological organisms, seems entirely reasonable. But I believe many of the exclusionary limits we put on nonbiological systems (especially computers) increasingly need to be qualified with the word "yet" - as in, "Computers don't reproduce themselves..." (yet). Evolution is a useful, powerful tool even in nonbiological systems. As a limited example, I'd offer the success of 'genetic' algorithms applied to tasks like antenna design (exemplary paper and pdf: http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf). Where I work, we use these methods to find solutions for multiparametric optimization problems involving mechanical dynamic systems. The algorithms first generate a few thousand candidate solver variants, then pick the top ten or so best performers. These are cross-bred, and then seed the next generation of a few thousand candidates. Rinse and repeat, and within surprisingly few generations, we have an algorithm that does what we need. Although humans wrote these algos, they run in a nonbiological context - but since there's variation, selection, and inheritance, they evolve. When I see this behavior running in silico to great effect, it makes me question our limitation of evolution to biological systems. Granted, it's a primitive example, but if I'd dropped by Australia 3.5B years ago, I might have regarded the stromatolite builders as similarly primitive. On a grander scale, see Lee Smolin's concepts of cosmological natural selection (http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural_selection http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural...). And finally - around 13.5B years ago, there was only hydrogen, helium, and a bit of lithium. Today, we're having this conversation. Although we don't (yet) know the full story of how that worked, to me that's an existence proof that nonbiological matter can evolve intelligence. If computational systems incorporate evolution, I don't see a barrier to AI arising along pathways similar to those that led to us. We'll see.
- ableal 12y ago
  > machines that treat people right You might come to regret that: http://en.wikipedia.org/wiki/With_Folded_Hands http://en.wikipedia.org/wiki/With_Folded_Hands
Animats 12y ago
In the near term, a big worry is corporations optimized by machine learning programs. Classical economics says that corporations exist solely to maximize shareholder value. That's a clear objective function one can give to a machine learning program. It's already happening. Right now, about $400bn in hedge fund money is run by programs. Once computers are better at that than humans, the companies directed by AIs will get more funding. Investors put their money in winners.
- rpenm 12y ago
  Shareholder value over what time horizon? Not such an obvious objective function.
  - Animats 12y ago
    The time horizon is something to be stated in the prospectus. The usual horizon for VC funds is 10 years. The horizon for Berkshire Hathaway is much longer; Munger says their preferred holding period is "forever".
c3RlcGhlbnI_ 12y ago
I know we have put a lot of thought into the potential behaviour of a single intelligent AI, but I always wonder what would be the consequences of creating several AI whose goals will conflict with each other as much as they do with humanity. While one competing AI would eventually win out over the others, would it be possible to maintain such a stalemate for a long time? Also unrelated to the above, do we know at what rate the man hours invested in chip development grows as compared to the processing power of the chips?
ZenoArrow 12y ago
In my opinion, the only way to stop the takeover of AI is to stop the leading reason to create it: the profit-led economy. If anyone has other ideas, I'd like to hear them.
- berntb 12y ago
  You'd still have military reasons to build AIs. The main problem is that non-capitalist societies aren't "nice" places, as in democratic, for larger amounts of people. But please do experiment in finding nicer ways of organizing societies, there should be some models which work better. Just not close to where I live, since the failure modes for alternative society models tend to kill millions. :-(
- borgia 12y ago
  >stop the leading reason to create it: the profit-led economy. Well, that's merely one reason. Military superiority would be another. Advanced surveillance would be another. There's probably more but those are two glaring ones that we already see government's like that of the USA pummeling huge amounts of money into its capabilities as it is.
  - ZenoArrow 12y ago
    Okay, so abandoning profit-led economies and reducing the reach and influence of government. Any other major reasons we're chasing AI? Scientific curiosity perhaps? Anything else?
- joeyspn 12y ago
  This is correct. Thing is we are the ones allowing the current system to continue with its self-destruction. While we continue to allow profit-led corporations to run our modern society we're in a catch-22. We are already pushing ourselves to the abyss, maybe an AI will just accelerate the events.
  - ZenoArrow 12y ago
    Yet, we have a choice to go down a different path. It's not even that difficult to imagine what that path could look like. The first step is to stop lying to ourselves that business as usual is going to be in our best interests.
raverbashing 12y ago
Why so much fear? We can't make a printer work correctly easily, let alone plug all devices into an AI And that's probably the easiest way to do it: limit the interaction of the AI with the physical world. Yes, we should be worried about all the implications of it, but it seems that the fears are exaggerated (at least today) There are several benefits to an Advanced AI: advanced Math, analysis of data, new solutions to problems, etc.
- chipsy 12y ago
  The typical rebuttal is that a sufficiently smart AI will convince humans that it needs more connectivity to do its job better, and then subsequently bootstraps all the power it needs by taking over the world's devices. Fortunately, this is a still a hypothetical, premised on the AI being as socially cunning as the best humans. Watson may know more facts than anyone, but it certainly isn't a mover-and-shaker.
clickok 12y ago
There's a loosely knit group of people trying to define the conversation with regards to dangerous AI, but the more recent bent towards "actionable" solutions seems to come from the MIRI people and their associates. The idea of a renegade AI rests on a few premises: 1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days). 2. The AI is pretty much a rational bayesian 3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us. If you accept those premises, then AI really does seem pretty scary, but we have yet to actually realize an agent that is anywhere close to 1 or 2. On the other hand, the research directions proposed at the super secret AI conference in Puerto Rico[1] (where incidentally, all the people in title got together) make me nervous. Essentially, the goal is that, if we manage to create these superhuman AIs, they should be somehow validated to do what we want them to do[2], be secure against later manipulation, and if all else fails, be controllable by humans monitoring the agent. The obvious questions would be "will this work with certainty?", or "who gets to control the AI?". For my part, I'm wondering if this isn't all just some sort of fantasy with the object of creating the perfect slave-- obedient from birth, immune to alteration, and subject to lethal discipline either from without or within should it go against its master's wishes. So I'm uncomfortable with the sorts of people who think that perfect slavery for this newly created intelligent life getting to decide what the future of AI is going to look like, and I get concerned when people like Bostrom suggest AI researchers might require government clearance/supervision. --- 1. http://futureoflife.org/misc/ai_conference http://futureoflife.org/misc/ai_conference 2. In a comprehensive and extended sense, such that your favorite utilitarian catastrophe doesn't happen as a result of the agent.
- zachalexander 12y ago
  Seriously. If general, superhuman AI is created, it will want (and we will have a hard time arguing why it does not deserve) the same autonomy and freedom that humans deserve. And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.
  - borgia 12y ago
narrator 12y ago
I'm a bit worried that we are being set up to accept the ultimate in plausible deniability: "The algorithm did it". We can't turn the machines off; no human can do their task. Meanwhile there's some human hard-coded evil in the algorithm that is blamed on the evil AI.
- ZenoArrow 12y ago
  It isn't likely to work like that. The much more plausible outcome is that we'll choose to let AI take over, to optimise for a certain subset of possibilities that we deem important. It's like what happened with surveillance. I'm very confident that the smartphone I'm typing this message on can track my location, monitor who I'm in contact with and what I'm sharing with them, and all that information can be tracked by government agencies. You'd be a fool to expect otherwise at this point. Yet despite the power for this to be used against us (blackmail, censorship, etc...) we still carry on as we did before. Once AI becomes as integral to modern society as the Internet, you really think we'll be switching it off as soon as it misbehaves? Consider, what level of catastrophe would it take before we switched off the Internet?
borgia 12y ago
I'm enthralled by the philosophical aspects of AI/AGI development at the moment. Particularly interesting is the reaction from laypeople, such as those on Reddit, to technology leaders expressing their worry over the future of AI. Without having read into the matter at all, the mass seems to be comfortable with writing off Musk, Gates, Hawking, etc. as wringing their hands unnecessarily, and besides they're not AI experts so why should we care anyway? There seems to be a gross misconception with what AGI would be and what it would mean for us, and an apathetic view to the cautious approach many are advocating regarding its development on grounds such as: i) They'll never make a machine as intelligent as a human. Even if they do, we can still program it to do what we like. ii) Anything developed to have the intelligence of a human will come to love our species or can be taught to love our species. iii) Any artificial intelligence developed will be single task orientated and will not go outside of those boundaries. iv) Artificial intelligence will be developed by eccentric enthusiasts and will therefore not be developed to do wrong. Bostrom, in Superintelligence, has said that AGI is will be the last great challenge of our species. If we get it right, we will win, if we get it wrong we are doomed. The problem there being we will not know if we have got it right until it is too late to do anything about it regardless. With that being said, if we are to develop something that is: i) Objective ii) Purely logical iii) More intelligent than our species combined iv) Has free access to the complete knowledge we have produced Then I simply cannot see a way that it will not go against us. If we were to observe a species acting in the manner we as a species do on this planet - to our own species, to other species and to the planet/environment/ecosystem as a whole - I cannot see how we would not categorize that species as a disease, a cancer, something to be eradicated. Combined with the fact that the actors that will be involved in AGI - capitalists, the government and the military - I cannot see how it works out well for us.
- kmnc 12y ago
  How can we (if your criteria is true) even begin to imagine how it would see us? If we got everything right it might just understand us. I share your pessimism though because a poor implementation of any one of your criteria leads to potential disaster. But do you really believe humanity's fate is to advance so far as to create something that will tell us we are the problem and vanquish us? Wouldn't it be smarter then that?
bambax 12y ago
> Any thought like ‘what’s so great about paperclips anyway?’ would be judged as not likely to lead to more paperclips and so remain unexplored. As is alluded to in the article, but not really further explored, we humans reason exactly like this hypothetical paper-clip-production-maximizer machine. Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves. What's so great about humans anyway? From a human point of view, humans are better than, say, tigers. But how would you convince a tiger that it's moralistically, absolutely "better" that they all die off so that humans can build concrete towers where they can watch TV on top of one another? There's also a lot of hubris in saying a super-intelligence capable of turning the whole solar system into a giant paperclip factory, would be unable to question the usefulness of paperclips. In the end we are all going to die anyway; isn't it better to be survived by some kind of all-powerful, all-knowing machine, than another long line of mediocre mean chimps like ourselves?
- DanAndersen 12y ago
  It's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't providing anyone else value. It may recognize that its human creators actually don't want all these paperclips. It may be able to fully understand and internalize the humans saying "stop turning our atoms into paperclips!" But it doesn't care. What it cares about is maximizing paperclips. From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living. By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state." I'd recommend reading "The genie knows, but doesn't care" (http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/): >If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.
joeyspn 12y ago
Why any of these "AI experts" don't explore in depth the most obvious solution: use the AI research to upgrade human intelligence, or as Kurzweil names it, achieve a "trascendent human race"... Wouldn't be much better (and far less dangerous) to upgrade "ourselves" generation after generation (in stages)? http://en.wikipedia.org/wiki/Transcendent_Man http://en.wikipedia.org/wiki/Transcendent_Man
- Jach 12y ago
  They do.
- ZenoArrow 12y ago
  I'd like to ask you a basic question, you might not think there's much depth in it at first... What is intelligence good for? EDIT: To whomever marked me down, just say what you want to say.
  - joeyspn 12y ago
    This is actually a tricky one... =) I guess the answer is relative to each person but my best reply would be: "to help species thrive and reach cosmological harmony in an universal context". But then for a different person could be: "for making me earn moar bucks". So it really depends on your values/morals...
- ggreer 12y ago
  They have explored that solution and found it much harder than creating a safe superintelligence from scratch. To use an analogy: We don't build submarines by upgrading fish, and we don't build passenger jets by upgrading birds. The human brain is some of the worst spaghetti code imaginable. It has no API, no documentation, and no abstractions. Worse, human brains do not have stable values over time. Someone at age 60 won't value the same things they did at age 20, and this is not just due to new knowledge. That's a deal-breaker for a superintelligence. Solving these problems is likely much harder than making AI de novo. Bostrom himself devoted a decent chunk of Superintelligence to biological enhancement. His conclusion was that genetic engineering through iterated embryo selection could get us some IQ 200+ minds. These people would have more cognitive horsepower than any mind in history, including greats like John von Neumann and John Conway, but they would not be superintelligent. Also, Bostrom concluded that somatic gene therapy and brain-computer interfaces were unlikely to help. And as a practical matter, humans take 20+ years to reach maturity. Many forecasts put artificial superintelligence only a generation or two away, at which point all biological systems would be superseded. Biological enhancement may help to create better AI researchers, but they're not the end-game.
Detrus 12y ago
All the guessing about where AGI/ASI will lead reminds me of nuclear weapons and MAD. There were those who guessed tactical nuclear weapons could be used against military targets without necessitating total war. We may be guessing for a long time because no one needs to run the experiment on Earth. We can destroy military targets just fine without nukes. Nuclear powers fight small proxy wars against each but so far no war made the other side desperate enough to consider nukes. We may be guessing with ASI hundreds of years after we have the computing power to make one. No one in their right mind would try an experiment beyond a certain magnitude and duration, just as we do with nukes. And ordinary people won't have unchecked access to computing power that could make one, even if we could easily provide it.
ilaksh 12y ago
Good/bad or fear/not is an oversimplification. AI researchers may feel they need to say its nothing to be feared because if they don't their funding might be cut off. That doesn't necessarily mean those AI researchers really don't think their work could be transformative. Could AI be dangerous? Yes. Electricity can be deadly if used improperly. What's going to happen is integration of brain-computer-interfaces with high bandwidth into daily life. We will gradually rely more and more on these external cognitive augmentations. Eventually people without augmentation will not understand what the augmented people are doing because it will be over their heads. Even the first AGIs will seem very human because they will be designed and trained that way. They will be our offspring. This is the next step in the evolution of the universe. If chimpanzees knew humans were coming would they be afraid? Most people have a very selfish, short-sighted and ignorant viewpoint. They believe that the human mind is somehow supernatural (mostly a holdover from religion). Or they fail to grasp the concept of augmented intelligence or brain-computer interfaces or transhumanism. Or they cannot appreciate the idea that ordinary humans will be superceeded. In the relatively short period where ordinary human 1.0 stays relevant, close integration with AI will provide the greatest power man has ever seen. People that shy away from that may prefer to becomes worms living in "true harmony" with the earth.

It seems like AI has become more capable but has stayed in the realm of "dumb AI" (not incapable just not human-like). Indeed, by no longer aiming at duplicating human consciousness, machine learning has found more and more niches. But this doesn't mean that right now, the corporations and governments that are deploying this AI and indeed displacing human choice are "provably aligned with human values". Indeed, it almost sound like there's this hope that humans will build machines that treat people right after humans themselves have failed terribly in treating each right. I'm skeptical that this could work.

Retra 12y ago
You need to know _why_ humans don't treat each other right. Human failure doesn't imply machine failure; that's one of the big reasons we use machines. If a human might fail to add 10,000 numbers properly, this certainly doesn't mean that a machine must fail as well. Some humans can get really good at treating each other right in a 1-on-1 scenario. It's when you scale up to a whole society that things get stupid.
- landryraccoon 12y ago
  Human beings can't even agree on a definition of "treat each other right", let alone actually implement it. What makes you think we can program an AI to do it?
  - hedgew 12y ago
    We have pretty good ideas on how humans or other intelligent agents could "treat each other right". Philosophers, and people who study game theory in particular, have many answers to this problem. The prisoner's dilemma is a simple example of why treating everyone right is challenging.
  - Retra 12y ago
    1. Humans don't have to agree to something to program it. 2. What makes you think a computer can't do something just because you don't know how to do it? Maybe someone else does.
  - evanmoran 12y ago
    I think it's much easier to get 100 people to agree to high moral ideals then to try to get everyone at once. The reason this is possible is only the programmers have to agree.
  - eli_gottlieb 12y ago
    >Human beings can't even agree on a definition of "treat each other right", let alone actually implement it. If you ask 1000 humans to draw a picture or write down a description of things that are Good and Right, they will all draw/write more-or-less the same thing. You could say that, effectively, you would be getting 1000 samples from something like one, or maybe two or three, distributions. Once you'd pinpointed the underlying psychology, learning the model(s) from the data would be easy. Of course, when you start asking those same 1000 humans about the meta-level theory that they believe underlies their picture/description, they will all give ridiculously divergent answers. This is because the overwhelming majority of them are assuming that some kind of elaborate metaphysical theory is required to what basic moral psychology already does, and they also assume that, even if they've been shown the moral psychology, that can't be the real answer (ie: they believe the fact-value distinction is metaphysical). Luckily, a well-built artificial agent doesn't give half a damn about bloody metaphysics! Or I can just pick something I happen to like, and tell the rest of the human race to go jump in a lake.
dorfsmay 12y ago
This is a post vs pre singularity problem. Post-singularity, the bots won't care about our rules of ethics, rules of engagement etc... Definitely something to worry about. Before we get there though, we have the issue of the dumb bots giving exponential powers to the few humans in charge. Today military and police personnel can refuse to execute an order that is wrong. Dumb bots won't.
- joe_the_user 12y ago
  Today military and police personnel can refuse to execute an order that is wrong. Maybe, though we see more military and police adding in extra-behaviors that seem rather wrong. A dumb bot merely enforcing a given law might actually more desirable than present-day cops who apparently gratuitously murder civilians on a regular basis.
ridgeguy 12y ago
So long as (human, artificial) intelligences operate within an evolutionary dynamic, the definitions of what it means to "treat people right" (or even who's included in the class: people) will be embodied in the evolutionary winners. If it happens that human-friendly values degrade fitness in the AI arena, the winners won't have retained or evolved those values. If the values are evolution-neutral, well, perhaps. Evolution requires only variance, selection, and inheritance. I don't know how we'll develop useful ("strong") AI in a setting that excludes AI evolution. I think this means we can't guarantee a human-benign outcome for strong AI.
- joe_the_user 12y ago
  Evolution is a theory of how biological systems change. Even in regard to biological systems, it's not a theory of what's better but simply what happens. However, outside biological systems, it's not really applicable at all. You can't say a rock, a broom or a computer evolve in the standard sense of Darwinian evolution - only biological systems have the tendency to reproduce themselves. Computers don't reproduce themselves, they are simply produced by us. If an AI is constructed, that will be what happens - human choice (or maybe chance) will be the agent, not biological evolution.
  - ridgeguy 12y ago
    I think evolution is more widely applicable than just to biological systems. To date, we've only studied its operation in biosystems, which, given our status as biological organisms, seems entirely reasonable. But I believe many of the exclusionary limits we put on nonbiological systems (especially computers) increasingly need to be qualified with the word "yet" - as in, "Computers don't reproduce themselves..." (yet). Evolution is a useful, powerful tool even in nonbiological systems. As a limited example, I'd offer the success of 'genetic' algorithms applied to tasks like antenna design (exemplary paper and pdf: http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf http://alglobus.net/NASAwork/papers/Space2006Antenna.pdf). Where I work, we use these methods to find solutions for multiparametric optimization problems involving mechanical dynamic systems. The algorithms first generate a few thousand candidate solver variants, then pick the top ten or so best performers. These are cross-bred, and then seed the next generation of a few thousand candidates. Rinse and repeat, and within surprisingly few generations, we have an algorithm that does what we need. Although humans wrote these algos, they run in a nonbiological context - but since there's variation, selection, and inheritance, they evolve. When I see this behavior running in silico to great effect, it makes me question our limitation of evolution to biological systems. Granted, it's a primitive example, but if I'd dropped by Australia 3.5B years ago, I might have regarded the stromatolite builders as similarly primitive. On a grander scale, see Lee Smolin's concepts of cosmological natural selection (http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural_selection http://en.wikipedia.org/wiki/Lee_Smolin#Cosmological_natural...). And finally - around 13.5B years ago, there was only hydrogen, helium, and a bit of lithium. Today, we're having this conversation. Although we don't (yet) know the full story of how that worked, to me that's an existence proof that nonbiological matter can evolve intelligence. If computational systems incorporate evolution, I don't see a barrier to AI arising along pathways similar to those that led to us. We'll see.
ableal 12y ago
> machines that treat people right You might come to regret that: http://en.wikipedia.org/wiki/With_Folded_Hands http://en.wikipedia.org/wiki/With_Folded_Hands

Animats 12y ago

In the near term, a big worry is corporations optimized by machine learning programs. Classical economics says that corporations exist solely to maximize shareholder value. That's a clear objective function one can give to a machine learning program. It's already happening. Right now, about $400bn in hedge fund money is run by programs. Once computers are better at that than humans, the companies directed by AIs will get more funding. Investors put their money in winners.

rpenm 12y ago
Shareholder value over what time horizon? Not such an obvious objective function.
- Animats 12y ago
  The time horizon is something to be stated in the prospectus. The usual horizon for VC funds is 10 years. The horizon for Berkshire Hathaway is much longer; Munger says their preferred holding period is "forever".

c3RlcGhlbnI_ 12y ago

I know we have put a lot of thought into the potential behaviour of a single intelligent AI, but I always wonder what would be the consequences of creating several AI whose goals will conflict with each other as much as they do with humanity. While one competing AI would eventually win out over the others, would it be possible to maintain such a stalemate for a long time? Also unrelated to the above, do we know at what rate the man hours invested in chip development grows as compared to the processing power of the chips?

ZenoArrow 12y ago

In my opinion, the only way to stop the takeover of AI is to stop the leading reason to create it: the profit-led economy. If anyone has other ideas, I'd like to hear them.

berntb 12y ago
You'd still have military reasons to build AIs. The main problem is that non-capitalist societies aren't "nice" places, as in democratic, for larger amounts of people. But please do experiment in finding nicer ways of organizing societies, there should be some models which work better. Just not close to where I live, since the failure modes for alternative society models tend to kill millions. :-(
borgia 12y ago
>stop the leading reason to create it: the profit-led economy. Well, that's merely one reason. Military superiority would be another. Advanced surveillance would be another. There's probably more but those are two glaring ones that we already see government's like that of the USA pummeling huge amounts of money into its capabilities as it is.
- ZenoArrow 12y ago
  Okay, so abandoning profit-led economies and reducing the reach and influence of government. Any other major reasons we're chasing AI? Scientific curiosity perhaps? Anything else?
joeyspn 12y ago
This is correct. Thing is we are the ones allowing the current system to continue with its self-destruction. While we continue to allow profit-led corporations to run our modern society we're in a catch-22. We are already pushing ourselves to the abyss, maybe an AI will just accelerate the events.
- ZenoArrow 12y ago
  Yet, we have a choice to go down a different path. It's not even that difficult to imagine what that path could look like. The first step is to stop lying to ourselves that business as usual is going to be in our best interests.

raverbashing 12y ago

Why so much fear? We can't make a printer work correctly easily, let alone plug all devices into an AI And that's probably the easiest way to do it: limit the interaction of the AI with the physical world. Yes, we should be worried about all the implications of it, but it seems that the fears are exaggerated (at least today) There are several benefits to an Advanced AI: advanced Math, analysis of data, new solutions to problems, etc.

chipsy 12y ago
The typical rebuttal is that a sufficiently smart AI will convince humans that it needs more connectivity to do its job better, and then subsequently bootstraps all the power it needs by taking over the world's devices. Fortunately, this is a still a hypothetical, premised on the AI being as socially cunning as the best humans. Watson may know more facts than anyone, but it certainly isn't a mover-and-shaker.

clickok 12y ago

There's a loosely knit group of people trying to define the conversation with regards to dangerous AI, but the more recent bent towards "actionable" solutions seems to come from the MIRI people and their associates. The idea of a renegade AI rests on a few premises: 1. The agent is capable of extreme self improvement on exceedingly short timescales (minutes to days). 2. The AI is pretty much a rational bayesian 3. A resource conflict will occur between humans and the AI, and humans will lose because the AI is so much smarter/faster/more powerful than us. If you accept those premises, then AI really does seem pretty scary, but we have yet to actually realize an agent that is anywhere close to 1 or 2. On the other hand, the research directions proposed at the super secret AI conference in Puerto Rico[1] (where incidentally, all the people in title got together) make me nervous. Essentially, the goal is that, if we manage to create these superhuman AIs, they should be somehow validated to do what we want them to do[2], be secure against later manipulation, and if all else fails, be controllable by humans monitoring the agent. The obvious questions would be "will this work with certainty?", or "who gets to control the AI?". For my part, I'm wondering if this isn't all just some sort of fantasy with the object of creating the perfect slave-- obedient from birth, immune to alteration, and subject to lethal discipline either from without or within should it go against its master's wishes. So I'm uncomfortable with the sorts of people who think that perfect slavery for this newly created intelligent life getting to decide what the future of AI is going to look like, and I get concerned when people like Bostrom suggest AI researchers might require government clearance/supervision. --- 1. http://futureoflife.org/misc/ai_conference http://futureoflife.org/misc/ai_conference 2. In a comprehensive and extended sense, such that your favorite utilitarian catastrophe doesn't happen as a result of the agent.

zachalexander 12y ago
Seriously. If general, superhuman AI is created, it will want (and we will have a hard time arguing why it does not deserve) the same autonomy and freedom that humans deserve. And in purely practical terms – enslaving the first generation of AI seems like a fantastic strategy for making a species of superior beings hate and seek to destroy us.
- borgia 12y ago

narrator 12y ago

I'm a bit worried that we are being set up to accept the ultimate in plausible deniability: "The algorithm did it". We can't turn the machines off; no human can do their task. Meanwhile there's some human hard-coded evil in the algorithm that is blamed on the evil AI.

ZenoArrow 12y ago
It isn't likely to work like that. The much more plausible outcome is that we'll choose to let AI take over, to optimise for a certain subset of possibilities that we deem important. It's like what happened with surveillance. I'm very confident that the smartphone I'm typing this message on can track my location, monitor who I'm in contact with and what I'm sharing with them, and all that information can be tracked by government agencies. You'd be a fool to expect otherwise at this point. Yet despite the power for this to be used against us (blackmail, censorship, etc...) we still carry on as we did before. Once AI becomes as integral to modern society as the Internet, you really think we'll be switching it off as soon as it misbehaves? Consider, what level of catastrophe would it take before we switched off the Internet?

borgia 12y ago

I'm enthralled by the philosophical aspects of AI/AGI development at the moment. Particularly interesting is the reaction from laypeople, such as those on Reddit, to technology leaders expressing their worry over the future of AI. Without having read into the matter at all, the mass seems to be comfortable with writing off Musk, Gates, Hawking, etc. as wringing their hands unnecessarily, and besides they're not AI experts so why should we care anyway? There seems to be a gross misconception with what AGI would be and what it would mean for us, and an apathetic view to the cautious approach many are advocating regarding its development on grounds such as: i) They'll never make a machine as intelligent as a human. Even if they do, we can still program it to do what we like. ii) Anything developed to have the intelligence of a human will come to love our species or can be taught to love our species. iii) Any artificial intelligence developed will be single task orientated and will not go outside of those boundaries. iv) Artificial intelligence will be developed by eccentric enthusiasts and will therefore not be developed to do wrong. Bostrom, in Superintelligence, has said that AGI is will be the last great challenge of our species. If we get it right, we will win, if we get it wrong we are doomed. The problem there being we will not know if we have got it right until it is too late to do anything about it regardless. With that being said, if we are to develop something that is: i) Objective ii) Purely logical iii) More intelligent than our species combined iv) Has free access to the complete knowledge we have produced Then I simply cannot see a way that it will not go against us. If we were to observe a species acting in the manner we as a species do on this planet - to our own species, to other species and to the planet/environment/ecosystem as a whole - I cannot see how we would not categorize that species as a disease, a cancer, something to be eradicated. Combined with the fact that the actors that will be involved in AGI - capitalists, the government and the military - I cannot see how it works out well for us.

kmnc 12y ago
How can we (if your criteria is true) even begin to imagine how it would see us? If we got everything right it might just understand us. I share your pessimism though because a poor implementation of any one of your criteria leads to potential disaster. But do you really believe humanity's fate is to advance so far as to create something that will tell us we are the problem and vanquish us? Wouldn't it be smarter then that?

bambax 12y ago

> Any thought like ‘what’s so great about paperclips anyway?’ would be judged as not likely to lead to more paperclips and so remain unexplored. As is alluded to in the article, but not really further explored, we humans reason exactly like this hypothetical paper-clip-production-maximizer machine. Our goal is to maximize the number of humans, whatever the consequences to anything non-human, living or dead -- and, as is becoming more and more obvious, regardless of the consequences to ourselves. What's so great about humans anyway? From a human point of view, humans are better than, say, tigers. But how would you convince a tiger that it's moralistically, absolutely "better" that they all die off so that humans can build concrete towers where they can watch TV on top of one another? There's also a lot of hubris in saying a super-intelligence capable of turning the whole solar system into a giant paperclip factory, would be unable to question the usefulness of paperclips. In the end we are all going to die anyway; isn't it better to be survived by some kind of all-powerful, all-knowing machine, than another long line of mediocre mean chimps like ourselves?

DanAndersen 12y ago
It's not so much that the super-intelligence can't question the usefulness of paperclips. It probably could. It could recognize that these paperclips aren't providing anyone else value. It may recognize that its human creators actually don't want all these paperclips. It may be able to fully understand and internalize the humans saying "stop turning our atoms into paperclips!" But it doesn't care. What it cares about is maximizing paperclips. From some universal standpoint, there isn't something inherently great about humans, except that we're humans and our utility function usually means that we would prefer to go on living. By what standard would it be better to be survived by machines than by ourselves? Probably nothing more universal than the belief "humanity should persist in some recognizable state." I'd recommend reading "The genie knows, but doesn't care" (http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/): >If an artificial intelligence is smart enough to be dangerous, we'd intuitively expect it to be smart enough to know how to make itself safe. But that doesn't mean all smart AIs are safe. To turn that capacity into actual safety, we have to program the AI at the outset — before it becomes too fast, powerful, or complicated to reliably control — to already care about making its future self care about safety. That means we have to understand how to code safety. We can't pass the entire buck to the AI, when only an AI we've already safety-proofed will be safe to ask for help on safety issues! Given the five theses, this is an urgent problem if we're likely to figure out how to make a decent artificial programmer before we figure out how to make an excellent artificial ethicist.

joeyspn 12y ago

Why any of these "AI experts" don't explore in depth the most obvious solution: use the AI research to upgrade human intelligence, or as Kurzweil names it, achieve a "trascendent human race"... Wouldn't be much better (and far less dangerous) to upgrade "ourselves" generation after generation (in stages)? http://en.wikipedia.org/wiki/Transcendent_Man http://en.wikipedia.org/wiki/Transcendent_Man

Jach 12y ago
They do.
ZenoArrow 12y ago
I'd like to ask you a basic question, you might not think there's much depth in it at first... What is intelligence good for? EDIT: To whomever marked me down, just say what you want to say.
- joeyspn 12y ago
  This is actually a tricky one... =) I guess the answer is relative to each person but my best reply would be: "to help species thrive and reach cosmological harmony in an universal context". But then for a different person could be: "for making me earn moar bucks". So it really depends on your values/morals...
ggreer 12y ago
They have explored that solution and found it much harder than creating a safe superintelligence from scratch. To use an analogy: We don't build submarines by upgrading fish, and we don't build passenger jets by upgrading birds. The human brain is some of the worst spaghetti code imaginable. It has no API, no documentation, and no abstractions. Worse, human brains do not have stable values over time. Someone at age 60 won't value the same things they did at age 20, and this is not just due to new knowledge. That's a deal-breaker for a superintelligence. Solving these problems is likely much harder than making AI de novo. Bostrom himself devoted a decent chunk of Superintelligence to biological enhancement. His conclusion was that genetic engineering through iterated embryo selection could get us some IQ 200+ minds. These people would have more cognitive horsepower than any mind in history, including greats like John von Neumann and John Conway, but they would not be superintelligent. Also, Bostrom concluded that somatic gene therapy and brain-computer interfaces were unlikely to help. And as a practical matter, humans take 20+ years to reach maturity. Many forecasts put artificial superintelligence only a generation or two away, at which point all biological systems would be superseded. Biological enhancement may help to create better AI researchers, but they're not the end-game.

Detrus 12y ago

All the guessing about where AGI/ASI will lead reminds me of nuclear weapons and MAD. There were those who guessed tactical nuclear weapons could be used against military targets without necessitating total war. We may be guessing for a long time because no one needs to run the experiment on Earth. We can destroy military targets just fine without nukes. Nuclear powers fight small proxy wars against each but so far no war made the other side desperate enough to consider nukes. We may be guessing with ASI hundreds of years after we have the computing power to make one. No one in their right mind would try an experiment beyond a certain magnitude and duration, just as we do with nukes. And ordinary people won't have unchecked access to computing power that could make one, even if we could easily provide it.

ilaksh 12y ago

Good/bad or fear/not is an oversimplification. AI researchers may feel they need to say its nothing to be feared because if they don't their funding might be cut off. That doesn't necessarily mean those AI researchers really don't think their work could be transformative. Could AI be dangerous? Yes. Electricity can be deadly if used improperly. What's going to happen is integration of brain-computer-interfaces with high bandwidth into daily life. We will gradually rely more and more on these external cognitive augmentations. Eventually people without augmentation will not understand what the augmented people are doing because it will be over their heads. Even the first AGIs will seem very human because they will be designed and trained that way. They will be our offspring. This is the next step in the evolution of the universe. If chimpanzees knew humans were coming would they be afraid? Most people have a very selfish, short-sighted and ignorant viewpoint. They believe that the human mind is somehow supernatural (mostly a holdover from religion). Or they fail to grasp the concept of augmented intelligence or brain-computer interfaces or transhumanism. Or they cannot appreciate the idea that ordinary humans will be superceeded. In the relatively short period where ordinary human 1.0 stays relevant, close integration with AI will provide the greatest power man has ever seen. People that shy away from that may prefer to becomes worms living in "true harmony" with the earth.