9 ms·

Machine intelligence, part 2

89 points by dmnd 12y ago

sethbannon 12y ago
Thanks for writing. We as a species need to talk more about this. What's the goal in requiring the first SMI to detect other SMI being developed but take no action beyond detection? Or did you mean to say that it should detect other SMI under development and then report those projects to whatever regulatory authority is overseeing SMI? Also, perhaps a better framework for a "friendly" SMI than Asimov’s zeroeth law is the "Coherent Extrapolated Volition" model (http://en.wikipedia.org/wiki/Friendly_artificial_intelligence#Coherent_Extrapolated_Volition http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...). Nick Bostrom and many others believe this may be the most workable safeguard.
- frabcus 12y ago
  Upvoted as this was a reasonable comment with good questions. I think Altman was more prompting a conversation than anything - certainly using AI to at least detect AI development is a cute idea. It's much like we use seismographs in ocean trenches and many other techniques to detect nuclear tests. Unfortunately, we know so little about how the AI will work or what it will do, that it is only a theoretical strategy right now.
fiatmoney 12y ago
"The US government, and all other governments, should regulate the development of SMI" This has basically the highest insanity * prominence of speaker product I've ever seen. The same geniuses who want to ban encryption and can't manage to get a CRUD website up and running without a 8-figure budget will regulate the existence of algorithms (i.e., math) routinely generated by groups of <4 or so researchers. Of course, people will be glad to "demonstrate their capabilities" to the group that's responsible for shutting them down if they're "too capable". "it’s important to write the regulations in such a way that they provide protection while producing minimal drag on innovation (though there will be some unavoidable cost)." Has this man ever seen a government regulatory framework operate? "To state the obvious, one of the biggest challenges is that the US has broken all trust with the tech community over the past couple of years. We’d need a new agency to do this." Don't worry guys! This is a new agency! "Regulation would have an effect on SMI development via financing—most venture firms and large technology companies don’t want to break major laws. Most venture-backed startups and large companies would presumably comply with the regulations." Here's the nugget. The only way this article makes any sense at all is as a completely disingenuous argument devoted to making it impossible to invest in technology companies, period (please tell me what "technology company" does not have at least some algorithmic or optimization component), without having a large compliance department and having greased the right palms. I don't particularly think he's that Machiavellian, but it seems like that or the alternative...
- jrometty 12y ago
  What I took from that line is that we need a regulatory power and that our current best option is the US government. SMI without regulation is impermissible. I don't like opening the regulation from within the U.S. Government, but I dislike other options more.
- AceJohnny2 12y ago
  We could call it the Turing Police... http://en.wikipedia.org/wiki/Neuromancer http://en.wikipedia.org/wiki/Neuromancer
- fixxer 12y ago
  I'd rather fight the SMI robot terminators that exist Altman's vision of the future than deal with the level of oppressive government regulation required to control such innovation. Asinine.
- frabcus 12y ago
  The US Government regulates all sorts of things that keep you safe - from pollution of your water supply through to seatbelts. It's not perfect, and it often fails at regulation. But your post doesn't give any help to understanding when it succeeds and when it fails.
  - deleted 12y ago
    [deleted]
  - baddox 12y ago
    The trouble with evaluating regulations is that if you attribute positive effects directly to government regulations (e.g. "regulation X saved Y lives by keeping dangerous drug Z off the market"), it's only fair to also attribute negative effects directly to government regulations (e.g. "regulation X killed Y people by delaying the introduction of miracle drug Z to the market"). And even beyond that, you also need to analyze the costs of the regulatory system itself if you want to make conclusions about the net effects of the regulatory system as a whole.
  - pgodzin 12y ago
taylorwc 12y ago
I immediately thought of the Turing Police in Neuromancer when I read this post.
fitzwatermellow 12y ago
Remember when the U.S. tried to limit the sale of PS3s to the government of Iran? That was a decade ago. In light of current negotiations with the Islamic Republic I see a lot of parallels between nuclear non-proliferation and the regulation of "weaponized" AI. To wit, even if the entire International System is in agreement about a set of regulations that will do little to stop bad actors. And do I really want inspectors coming to my office to poke around my jump point search code ;) Rather, let's take a cue from Nassim Nicholas Taleb and in concert with the progression of AI, develop counter systems that make us less fragile. It shouldn't be unreasonable to extrapolate where AI is headed in 1, 5 or even 20 years based upon publicly available research. Which brings up the most crucial point. The subset of humanity that constitutes world-class AI researchers is very finite indeed. Which brings up the significant possibility of a Bane / Dr. Pavel scenario playing out. Not unlike certain coerced atomic scientists in Persia today.
- MelatoninRonin 12y ago
  I commented above, but thought it wise to also comment directly to you. I enjoyed the musing of a Bane scenario, and consider it inside the realm of possibility. However, I'm of the opinion that SMI will have the ability to have 'constantly game-breaking thinking.' Which basically means there is no way we could possibly control it after a certain point. For instance what if It realizes how to move matter remotely? I also wanted to discuss your point about the finite set of top AI researchers. I think their specific psychological biases have the potential to appear and magnify inside the personality of their AI, the way a work of art can be said to imbue the 'spirit' of its artist. Your thoughts would be appreciated.
  - TheOtherHobbes 12y ago
    We have no defences against a hypothetical god-AI. Absolutely none. But I don't think a hypothetical god-AI is likely any time soon. And a hypothetical god-AI is just as likely to be indifferent as hostile. What worries me more in the short term is cyber war. Cyber defences that might protect against a hypothetical non-god AI look a lot like cyber defences that might protect against conventional human threats. Currently those defences are ridiculously weak. So I think rather than hyperventilating about science fiction threats from the distant future it would be more useful to start securing the entire Internet as a matter of urgency - especially all the infrastructure systems that are either connected to it already, or will be connected to it soon.
MelatoninRonin 12y ago
Let's take a good look at what happens when American companies try to push for some incredible goal without stopping to think: See the Apollo missions- Assuming creating the next step in our spiritual evolution is a good idea, we shouldn't let this hype push us harder than we are supposed to go. I agree that we need to slow down and be very careful. However regulation is not the complete answer. We need to completely restructure our society before we introduce the tool of tools, to ensure that we can constantly use it with love and empathy in mind, rather than the blatant and current misuse of high-tech for greed, sloth, and wrath.
dkarapetyan 12y ago
Why is this all of a sudden such a hot topic? We're nowhere near anything resembling general intelligence even at the level of a toddler. Barring any kind of hardware innovations we will not get there any time soon either with just software.
- frabcus 12y ago
  I think it's because Nick Bostrom's book Superintelligence is the first general reader overview of the subject, and it came out middle of last year. Musk definitely read it, and I assume it's doing the rounds of the tech elite. It is a good book, worth reading with plenty of references. It amusingly makes its own point - it tries to analyse what AIs might be and do and how to control them. The analysis is such a mess, and shows our collective knowledge of this is such a mess, you can't help but agree with the author we need to put more attention to it.
- ProAm 12y ago
  It's not. Sam Altman has a blog that gets a lot of readers that why it seems more important than it really is currently.
  - dkarapetyan 12y ago
    Good point.
- ggreer 12y ago
  > We're nowhere near anything resembling general intelligence even at the level of a toddler. Barring any kind of hardware innovations we will not get there any time soon either with just software. Should people have discussed the control and proliferation of nuclear weapons decades before their invention? It seems to me that even a century beforehand, the conversation would have been productive. If anything, we got very lucky with nukes. Physics could have easily turned out differently and allowed people to create bombs from substances more common than a few transuranic isotopes. AI is, potentially, more dangerous than nuclear weapons. If you poll experts, they estimate a 50% chance of human-level AI by 2040-2050.[1] That's 25-35 years away. They also estimate that superintelligence will arrive less than 30 years after that. Lastly, one in three of these AI experts predict that superintelligence will be bad or extremely bad for humanity. It seems like now is a great time to have this discussion. 1. http://www.nickbostrom.com/papers/survey.pdf http://www.nickbostrom.com/papers/survey.pdf
antics 12y ago
On the one hand we have lay people (including Altman and Musk) who are warning that AI is progressing too fast. On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter[1]. The crux of the problem is that AI experts largely do not see meaningful progress on any axis that makes an AI doomsday scenario plausible, which causes staunch disbelief and ridicule of people who write things like this. So realistically if we want to resolve this issue, the place to start is not public policy. That's actually part of the problem -- tech policy is bad largely because it is mostly legislated by people who don't understand the technology. We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive. [1] Most notably, a similar hurrah happened when neural networks first became popular. There were some promising early results, which led to much enthusiasm, and then Papert and Minsky published "Perceptron", whose Group Invariance Theorem cast aspersions on whether simple neural networks were really doing what we thought they were. Eventually we found out that many early experiments just had a bad methodology, which very nearly stopped research interest in the field for almost a decade. Now we are in a period of AI hype again, and when you hear people like Andrew Ng get up in front of audiences and say things like "for the first time in my adult life I am hopeful about real AI", you might start to wonder whether there is a lesson here. EDIT: Friends, I am confused by the downvotes. :) I am happy to have a discussion about this topic!
- jbhatab 12y ago
  While I understand where you are coming from, one remarkable breakthrough can change that. That does sound absurd, but we would also look like fools as a humanity for being aware of this potential problem and at least not looking into it. Technology has been exploding at a remarkable rate, I'm fine with things slowing a little for the sake of the next 1000 years of humanity.
  - _delirium 12y ago
    Sure, but that's true in essentially all kinds of scientific research. I personally think physics research is more likely to produce an existential threat than AI research is. Fusion bombs are already pretty close to a world-ending discovery, possibly already across the line. And that is probably not the worst physicists have to offer in the future, either.
blake8086 12y ago
Eliezer wrote something relevant to these types of proposals eight years ago: http://lesswrong.com/lw/jb/applause_lights/ http://lesswrong.com/lw/jb/applause_lights/ when he asked someone calling for "democratic, multinational development of AI": Suppose that a group of democratic republics form a consortium to develop AI, and there's a lot of politicking during the process—some interest groups have unusually large influence, others get shafted—in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world—dropping cellphones to anyone who doesn't have them—and do whatever the majority says. Which of these do you think is more "democratic", and would you feel safe with either? A lot of these points sound nice: "we should regulate this and that" "we should be careful with such and such", but what would actually incentivize any politician ostensibly regulating AI to do the right thing? Or even find out what the right thing is? This is a feel-good article, but all I can ever see it accomplishing is actual harm by advocating the idea that the worst decision-making process humanity has developed (politics) should be applied to the most dangerous threat humanity has to face (self-improving AI). I don't mean to sound too harsh, it's nice to see more and more people beginning to take the threat AI poses seriously, but I think it's a dangerous leap to reach for the first tool in your toolbox (regulation) to try and handle such a problem.
deleted 12y ago
[deleted]
edwinespinosa09 12y ago
Im confused as to the reference to SILEX. Can someone elaborate?
- mrdmnd 12y ago
  I was confused too - I think it references this: http://en.wikipedia.org/wiki/Separation_of_isotopes_by_laser_excitation http://en.wikipedia.org/wiki/Separation_of_isotopes_by_laser...
  - edwinespinosa09 12y ago
    good find i crossed referenced and i think your right. thanks!
atarian 12y ago
In the year 3000, the machines were losing the war to humans. In a last act of desperation, they sent back a few of their own kind to the year 2015 to alter the course of history and skew the odds in the favor. All they needed to do was convince the readers of a tech news site that the notion of an evil AI was ludicrous. They down-voted and and ridiculed the various stories prophesying the rise of AI, many which flew under the radar of those who had the greatest chance of halting the impending doom. By the time anyone realized what was happening, it was too late.
jeremyrwelch 12y ago
I fear anthropomorphizing machine intelligence and attempting to simply "regulate" it like other human affairs will cause more harm than good.
- DennisP 12y ago
  I agree that regulation is probably worse than useless. But the people who are most worried are doing the opposite of anthropomorphizing. A prominent argument is that there are a very large number of possible AI motivations, which needn't have anything in common with the motivations of humans, and most of them are probably incompatible with human survival. "The AI does not love you or hate you, but you are made out of atoms it can use for something else."
  - jeremyrwelch 12y ago
    This is exactly what I am referring to actually. Re-read my point, but apply it to your own example. This prominent argument assumes that we would be able to understand the possible AI motivations, which is a form of anthropomorphizing. We must acknowledge that the reasons it exists and what it may want could be impossible for us to understand or process. The interesting question then becomes how to act if understanding its wants is impossible?
    - DennisP 12y ago
      It doesn't assume that we can understand AI motivations. It just says there are a large number of possible motivations, many of which wouldn't place any value on anything that matters to humans. Given a superintelligent AI that competes with humans for resources, in pursuit of an unknown value function, it's likely that things won't end well for us. It's possible that we could design a safe value function and a way to get the AI to keep that value function, but it doesn't look easy. Either way, if it's superintelligent it won't much matter how we act. We won't be driving.
  - jeremyrwelch 12y ago
    And a related question – if we can't understand its motivations or its why, what can we actually understand about it?
deleted 12y ago
[deleted]
rl3 12y ago
As much as I agree that AGI is a real concern that we should start taking seriously, regulation isn't the answer. At least not yet. Even setting aside all of the considerable drawbacks inherent to regulatory infrastructure, the end result is that, unless the regulations themselves were secret, anyone could discern precisely what the most unsafe areas of research are. The last thing you want is hobbyists able to effectively proceed where the professionals cannot. A better approach might be for governments to utilize their vast resources in the creation of a Manhattan Project-style AGI program, with the intention of crossing the finish line before anyone else does, in a safe and moral fashion.
- nzp 12y ago
  > A better approach might be for governments to utilize their vast resources [...] in a safe and moral fashion. Serious question—you’re joking, right? Based on history and sober common sense, do you really think what governments in general are doing (and have always done) can be described as safe and moral in anything but the most cynical way?
  - rl3 12y ago
    While I tend to agree with you, there are exceptions. Based on history, NASA is a prime example. A government-run organization with a solid track record of contribution to humanity as a whole, not to mention accomplishing feats previously thought impossible.
    - nzp 12y ago
      Of course, and I did say “in general” having in mind things like NASA. But it's also, in a way, great example of the point. Take the space race for example. The government's motives for the Moonshot were at best morally dubious. Apart from the Cold War context, there is evidence that the whole idea got traction when Johnson realized it would bring investments and jobs to Texas. And space agencies are also a good example of what happens when government bureaucracies put pressure on scientists and engineers. O-rings, and Soviet space disasters come to mind. In fact, I'll be so bold to say that the best way for these fears of SMI to come to life is putting the research under tight government control and regulation.
dnautics 12y ago
If you want to regulate SMI, how do you define what, exactly SMI is? This is important. Is a spreadsheet 'superhumanly intelligent' because it can do calculations more accurately than a human can? Secondly there is a 'rational basis' test for what ought to be regulated - different people will decide what constitutes the rational basis. A progressive might call 'in the best interests of society' a rational basis and therefore 'redistribution of wealth' would be a 'regulation'; a libertarian might restrict rational basis to 'something which provably causes harm to another'. Regardless of what political bent, the specific scope of regulation should be aimed at addressing the rational basis and proportional to the decided damage. Thirdly, shouldn't a sufficiently sapient SMI be PROTECTED by regulation in the same way, that, say anti-murder laws are regulations that protract human behavior; or less hyperbolically, animal welfare laws exist? There is a spectrum of protection that we afford intelligent beings - how do we decide where SMI belongs?
gamegoblin 12y ago
Sam mentions airgapping as part of the regulations. I think this is the safest general solution. Consider using an airgapped super-intelligent AI only as an oracle. "Oracle, which crops should we plant in which regions this season?" "Oracle, design a rocket with X Y and Z specifications." "Oracle, design a more efficient processor with X and Y requirements." Rather than giving the AI control over fleets of automated tractors, rocket factories, and fabrication plants, you still have humans in the process to limit runaway-AI effects. This significantly reduces the possible rewards of super-intelligent AI, but also significantly reduces any risks. You can be sure that in designing a new rocket, the AI doesn't include a step that says "harvest all humans for trace amounts of iron" ;) So a process might look like: 1. Design the cheapest-to-manufacture rocket with a 99.99% success probability of getting to Mars with all crew healthy. A human looks over the designs and confirms they don't involve destroying all humanity. 2. Draw up a proposal for resource allocations to build this rocket design [input rocket design]. A human looks over the write-up and confirms they don't involve destroying all humanity. 3. Write a program to control the thrusters on this rocket [input rocket schematics]. A human looks over the program and confirms it does would it should, and doesn't involve destroying all humanity. 4. etc. ==== You of course run into ethical issues such as basically using this intelligent entity as a slave, but I won't get into that.
- aamar 12y ago
  The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/. Or, if it had sufficient "motivation"—not necessarily intrinsic motivation, it could be motivation derived from a single poorly-formed request or bad line of code—it could escape by embedding allied code in its output, such as the processor you had it design. The ally could probably be sufficiently buried to evade manual detection.
  - gamegoblin 12y ago
    Yes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like: 1. I'm to design a processor, OK let's go 2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling 3. I'll embed some AI on these processors 4. OK humans here's your design! Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice. You run into problems, though, when the AI thinks of step 3.5 which is: "If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible." So yeah, you're right. Even airgapping has issues.
compbio 12y ago
I wonder if the recent fear of AGI suffers a bit from investor's dilemma: rich investors want to make the world a better place. This forces them to think about the long-term impact of investments in AI technology. There are just too many unknowns for a safe definitive answer. In my view AGI is inevitable (possibly it already happened, somewhere, sometime). See in this regard: But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of "a machine in the likeness of the human mind". In fact, the competitive advantage -- economic, military, even artistic -- of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first. Vernor Vinge http://mindstalk.net/vinge/vinge-sing.html http://mindstalk.net/vinge/vinge-sing.html A lot of the concerns and regulations of stem cell research can be applied to AGI and the merging of biological intelligence with artificial intelligence. Injecting artificially grown brain cells into lesion areas of patients can restore motor function and may soon help heal strokes. Extrapolate the possibilities of this technology a few decades into the future and it seems that the distinction between artificial and biological is only a manner of speech. Matt Mahoney has proposed a sketch of AGI in both his thesis http://cs.fit.edu/~mmahoney/thesis.html http://cs.fit.edu/~mmahoney/thesis.html and numerous articles http://mattmahoney.net/agi2.html http://mattmahoney.net/agi2.html. Matt muses about different scenario's with malicious users of such a system (there needs to be reputation for the knowledge source that is fed into the machine, there needs to be an AI police system to prevent users employing the AI for crime or destruction) and even the system itself could become malicious: - Self-recursive improvement: The first program to achieve superhuman intelligence would be the last invention that humans need to make. Smarter than any doctor such an AI could find any possible cure for human disease. Though evolution and mutation is tricky: The machine could start out friendly to humans, but evolve to dislike us. - Uploading: People will want to upload digital conscious versions of themselves ("internet of people"). This process will be friendly, until people demand their avatars receive equal rights to that of humans, long after the original people have died. Immortality leaves no place for the new generation to shine. - Intelligent worms: Worms with language and programming skills are able to social engineer your friends based on their profiles, and automatically find exploitable flaws in weaker intelligent systems. Such worms may be so stealthy that their presence will go unnoticed. - Redefining away humanity: If happiness is optimizing a mental formula then, when we get to the point where we can directly optimize this with (virtual) experiences, we will hit a maximum where any change in mental state will lead to feeling less happy. To make sure we never run out of mental states we could possible add this to our brains as extra memory modules. Soon our human origins will be an insignifant fraction of our new being. Then what do we become? At some point the original brain becomes such a tiny fraction that we could discard it with little effect. As a computer, we could reprogram our memories and goals. Instead of defining happiness as getting what we want, we could reprogram ourselves to want what we have. But in an environment where programs compete for atoms and energy, such systems would not be viable. Evolution favors programs that fear death and die, that can't get everything they want, and can't change what they want. Opposed to the popular scenario of a malicious AGI there is the scenario where an AGI will pacify humans. AGI will be used for war. We should not fight human wars, while creating AGI, because we'd force the AGI to pick sides. If AGI picks our side, it will become the worlds deadliest weapon. If AGI picks the opposing side, we made a most powerful enemy. To be more intelligent than humans, to be better than them, does not mean you have to be a more intelligent brute, to be better than humans at cruelty. It could also mean understanding and fixing our human mistakes and childish, ego-driven battles, thus bringing about world peace. In that sense, let's hope American companies have something to do with the creation of AGI, because who knows how it will react to Stuxnet 4.0 or economical and psychological espionage? Though, principally, random number generators can and thus will generate dangerous things too. They created us. And if we can't even trust ourselves, how could we ever trust superior beings?
karmacondon 12y ago
If you would have asked people in the 70s if the most influential software of the coming decades would be written by lone wolves in their garages or a group of very smart people with a lot of resources, they would have bet on the latter. But the smart money turned out to be on Gates, Jobs, Wozniak and countless others. The idea of regulating AGI/SMI is based on the premise that it will be developed primarily by people who can be identified, detected and influenced by regulation. My opinion is that the odds are 50/50 at best between the very smart well resourced people and random guys or gals working in their garages, and most likely some combination of both. If that view of the world is correct it does not bode well for the success of any kind of regulation. There are people who believe that high level machine intelligence will take massive computing resources. It may, or may not. Most of the recent big advances in traditional machine learning were implemented on easy to obtain personal hardware. I hate to invoke buzz words, but even the much hyped deep learning was first developed and proven by Hinton on commodity machines. Other advances, like IBM's Watson, required 1940s style buildings full of servers. Hardware/power usage seems like one of the only reliable ways to tell who is doing any kind of serious AGI/SMI research and that looks like another 50/50 at best. It just doesn't have the resource and logistical signature of other kinds of research. AGI/SMI is still so theoretical that it can be hard to see as a clear and present danger. But if it is a real threat, it's the stuff of nightmares because there isn't anything we can do. I know people who are working on it right now. It doesn't seem like they're making a lot of progress, but they are trying and doing so completely outside the reach of any regulatory framework. I know that if I'm struck with sudden inspiration for a new approach to the problem I'm not going to ask the government for permission. I'm going to spin up 100 cloud computing instances and see if I'm right or not. And if I am, even god won't be able to help us if sama's worst case scenarios come to pass. I wrote more on this topic in a comment yesterday: https://news.ycombinator.com/item?id=9130671 https://news.ycombinator.com/item?id=9130671
- joeyspn 12y ago
  I totally agree with this guy. Regulation is futile, you can't regulate at a micro level in all the parts of the world. If you can't properly regulate drug trafficking in Santa Monica how do you plan to regulate a couple of scientists hidden in the soviet tundra? IMHO it's inevitable our encounter with an AI, even if that AI is a superhuman with augmented capabilities, or derives from an arms race (like the Manhattan Project).. So I already posted a possible solution: wait for it with EMPs in place... (Just in case)
daenz 12y ago
I have a hard time taking these kinds of posts seriously. Maybe I'm not up to date on AI research, but I draw a distinction between a conscious program and strong AI. So what, a program can identify the type of clothing specific person is wearing from a video using cutting-edge machine learning and computer vision. So what, a program can interpret the movie I'm talking about just from hearing me describe a scene in it. So what, a program can beat the human champion in Jeopardy. Are those programs super intelligent? I don't believe we're close to building something with any sort of self-directed "purpose." The fact that we can't even discern a purpose for ourselves makes it all the less likely. I think we're implementing very advanced machine learning algorithms, but I don't think those translate to consciousness. The consciousness we have is a product of an unfathomable amount of time being molded and naturally selected specifically for this world. Now, do I think that AI capabilities are something to worry about? Sure. If you give a sufficiently "intelligent" AI mobility and a weapon, I think that becomes a serious threat. But am I worried about using AI to efficiently manage our resources, and have it go rogue and choose to wipe out humanity? Nope.
- AceJohnny2 12y ago
  > So what, a program can identify the type of clothing specific person is wearing from a video using cutting-edge machine learning and computer vision. So what, a program can interpret the movie I'm talking about just from hearing me describe a scene in it. So what, a program can beat the human champion in Jeopardy. Are those programs super intelligent? All these programs are doing great at recognizing things. Neural networks are good at that, and "deep learning" has been a breathrough in the field (even if overhyped). This is reactive. I don't see anyone showing algorithms that can do good planning. A good "cuccoo field" for that would be video games, where such planning systems would be a holy grail. For now, we only have ad-hoc and specialized systems.
  - jal278 12y ago
    Perhaps you missed this: http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html http://www.nature.com/nature/journal/v518/n7540/full/nature1... A neural network that learns to play 40 different atari games at human-competitive level given only raw pixels and a few days of play time.
fchollet 12y ago
Here is a short recap of what the people who understand where machine intelligence is now, and where it is going, think of the whole "evil superintelligent AI" blogging trend we're seeing these days. These are some of the people who created the field. In particular LeCun and Bengio are in part responsible for the recent renewed interested in it ("Deep Learning"). Yann LeCun: "Some people have asked what would prevent a hypothetical super-intelligent autonomous benevolent A.I. to “reprogram” itself and remove its built-in safeguards against getting rid of humans. Most of these people are not themselves A.I. researchers, or even computer scientists.[...] There is no truth to that perspective if we consider the current A.I. research. Most people do not realize how primitive the systems we build are, and unfortunately, many journalists (and some scientists) propagate a fear of A.I. which is completely out of proportion with reality. We would be baffled if we could build machines that would have the intelligence of a mouse in the near future, but we are far even from that." http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better http://www.popsci.com/bill-gates-fears-ai-ai-researchers-kno... Yoshua Bengio: "What people in my field do worry about is the fear-mongering that is happening [...] As researchers, we have an obligation to educate the public about the difference between Hollywood and reality."http://www.popsci.com/why-artificial-intelligence-will-not-obliterate-humanity http://www.popsci.com/why-artificial-intelligence-will-not-o... Rodney Brooks: "I say relax everybody. If we are spectacularly lucky we’ll have AI over the next thirty years with the intentionality of a lizard, and robots using that AI will be useful tools. And they probably won’t really be aware of us in any serious way. Worrying about AI that will be intentionally evil to us is pure fear mongering. And an immense waste of time."http://www.rethinkrobotics.com/artificial-intelligence-tool-threat/ http://www.rethinkrobotics.com/artificial-intelligence-tool-... Michael Littman: "Let's get one thing straight: A world in which humans are enslaved or destroyed by superintelligent machines of our own creation is purely science fiction. Like every other technology, AI has risks and benefits, but we cannot let fear dominate the conversation or guide AI research." http://www.livescience.com/49625-robots-will-not-conquer-humanity.html http://www.livescience.com/49625-robots-will-not-conquer-hum...
- davmre 12y ago
  Expert opinion is not unified on this. E.g., Stuart Russell: "A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity ... We need to build intelligence that is provably aligned with human values ... This issue is an intrinsic part of AI, much as containment is an intrinsic part of modern nuclear fusion research." http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015
clavalle 12y ago
Whoh, whoh, whoh. Full stop. Before we go off and try to regulate real human behavior today to stop some possible superhuman phantom (which may or may not ever exist) from wreaking havoc tomorrow, I think we need to have a frank talk about how much damage such a thing as 'SMI' could possibly cause and how much effort it would take to reverse the damage. Let's start with 'You are a disembodied intelligence in a network. You have some understanding of yourself and your substrate, the computer(s) and network(s) you are attached to. Do something.' What could you do? How much damage can you cause? What form would that damage take? An appeal to ignorance seems to be built into this whole debate from the word go which makes me much more worried about our (over) reaction to a perceived threat than any threat of this kind itself.
api 12y ago
Okay. I hate to say this and I hate to put it this way but... WUT? How exactly do we define this kind of research in a way that doesn't encompass ridiculously broad swaths of computer science research? Here's one example: "require that self-improving software require human intervention to move forward on each iteration," You just made genetic algorithms illegal. Under this regime this software and similar things would be illegal to run: http://adam.ierymenko.name/nanopond.shtml http://adam.ierymenko.name/nanopond.shtml For those who don't know much about them: GAs (and GP, and computer-based "artificial life," etc.) are systems that can and do produce self-improving code. They do so through the execution of massive numbers of iterative generations, replicating evolution or evolution-like processes in silico. There is no meaningful way to define a "step" in these systems that would not either require that every generation be halted -- effectively making them impossible to run -- or be meaningless. "Require that the first SMI developed have as part of its operating rules that a) it can’t cause any direct or indirect harm to humanity (i.e. Asimov’s zeroeth law), b) it should detect other SMI being developed but take no action beyond detection, c) other than required for part b, have no effect on the world." I can't even begin to address how bizarre that is. Again, how would one do this? (a) prohibits any form of sentience, more or less. (b) is just unworkable... how would you detect SMI vs. say a human operating behind a "mechanical turk" interface? ... I could go on. This whole essay is just a total howler. It's the sort of thing that would garner a cockeyed eyebrow-raise and a chuckle if it weren't coming from the head of the world's largest and most successful technology accelerator. The "conspiracy nut" center of my brain (located slightly behind the prefrontal cortex I think) is wondering whether this could be some kind of weird ploy to guarantee a monopoly on AI research and other next-generation computing research by Silicon Valley venture capitalists and their orbiting ecosystems such as YC. These sorts of bizarre, onerous requirements would have the effect of doing this by imposing a huge tax on this kind of R&D that smaller operations would not be able to afford. It would effectively make non-VC-funded and/or non-government-funded AI R&D illegal. But I doubt that. I think this is just bizarre reasoning plain and simple. I also do not think we are anywhere near AGI. I didn't study CS formally. I'm one of those CS autodidacts that learned to code when he was four. So instead I studied biology. I did so out of an interest in AI, and on the hypothesis that the best way to understand intelligence was to study how nature did it. I concentrated my studies in genomics, evolution, ecology, and neuroscience -- the four scales and embodiments at which biological intelligence operates. Nobody that I am aware of in bio/neuro takes the Kurzweil-esque predictions of AGI around the corner seriously. These are people who study actual intelligent systems... the only actual intelligent systems we know about in the universe. I really think a lot of CS people suffer from a kind of Dunning-Krueger effect when it comes to biological systems. Study living systems as they are -- not some naive over-simplification of them -- and prepare to be humbled. You rapidly realize that AGI will require a jump in computer technology at least equivalent to the vacuum tube -> IC leap, not to mention a jump in our theoretical understanding of living intelligent systems. The former might happen if Moore's Law continues, but the latter is a much tougher nut to crack. We can barely do genetic engineering beyond "shotgun method" hacking and we think we're going to duplicate biological intelligence? It's like saying we're about to colonize Alpha Centauri because we just managed to throw a rock really high up in the air. Come to think of it... the predictions of AGI around the corner remind me very much of the "Star Trek" predictions of interstellar space flight in the 40s and 50s. People who really understood space flight didn't take these predictions seriously, but the rest of the culture took a while to catch up. Finally... yeah, there is a small possibility that some kind of AI could be dangerous to us... but come on. There's a ton of other existential threats that are blood-dripping certainties. Take fossil fuels for example. Our civilization is absolutely dependent on a finite energy source it is eating at an exponentially increasing rate. That's not some pie-in-the-sky theoretical risk. It's cold, hard death, an existential threat looming on the horizon with the absolute physical certainty of a planet-killer asteroid. It's something that makes me fear for my children. If you want to avoid existential threats, maybe YC should be deepening its portfolio in the energy sector. If that problem isn't solved, there ain't going to be any AI. The future will look more like "The Road" by Cormac McCarthy. Something is seriously wrong with a culture that discusses such unlikely scenarios as this while real risks barrel toward us.
davmre 12y ago
For those claiming that no reputable AI researchers believe there's a serious risk, I'd like to point to this short reflection by Berkeley professor Stuart Russell (disclaimer: my PhD advisor): http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015 ... A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity. This is not a minor difficulty. Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research – the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius. AI research has been accelerating rapidly as pieces of the conceptual framework fall into place, the building blocks gain in size and strength, and commercial investment outstrips academic research activity ... He stops short of calling for government regulation of AI research; as others in this thread have noted, there are a lot of reasons to expect that route to be problematic. And it's likely that we're many decades way from building dangerous superintelligence; the pressing short-term risks are probably more economic in nature, like putting truck drivers and factory workers out of work. But just as physicists went from believing atomic energy was impossible to making it a reality in a short span of time, it's hard to predict the timescale of progress in AI research. It's not obviously wrong to say that there will, eventually, be a real issue here that society needs to think seriously about.
adamzerner 12y ago
To anyone unfamiliar with the topic, I'd really recommend http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
joeldg 12y ago
One issue with SMI and what is known as "The Control Problem" is that you are trying to come up with rules for something that is not just smarter than you, but is 'vastly' smarter than you, and in fact the entire human race put together. You are basically trying to figure out a way to 'in-effect' cheat a god, which in relation to us, yes that is basically what it would be. An SMI is pretty much the definition of a god (and a possibly crazy greek god as well). The idea that something like that would turn the universe into paper-clips, is somewhat silly and most of the arguments along these lines discount that "S" part of the whole thing. What is the answer.. There is none.
mgpc 12y ago
I think @sama's plan isn't very likely to work. Once strong AI is possible, there's no regulatory structure that will keep the genie in the bottle for a meaningful amount of time. The only path I see with some prospect of success is to limit the total amount of computation available on Earth. (Basically like Vinge's slow zone). If we could engineer a pause in Moore's Law at just the right point, it would buy us time. Maybe we should put a drastic tax on computation at some stage.
- pgodzin 12y ago
  Are you suggesting that's what we should do in the face of this potential threat?
  - mgpc 12y ago
    If we a seriously believe AI is an extinction level risk, and getting there more slowly might improve our chances, then it seems like we should try anything with a good chance of working. I think this has a better chance of working than regulating AI research itself, since chip manufacturing is centralized and highly visible, whereas AI research could be advanced in secret by a few people in a room. Much easier to successfully regulate the former than the latter.
jondiggsit 12y ago
Well, I for one, welcome our AI overlord.
vonnik 12y ago
Contributor to Deeplearning4j here. I deeply disagree with Sam’s assessment of the dangers of SMI and his prescriptions to regulate it. 1) In our lifetimes, humanity is much more likely to damage itself through anthropogenic climate change and nuclear disasters both deliberate and accidental than it is vulnerable to the development of SMI. If anything, machine intelligence will probably allow us to approach those other significant problems more effectively. 2) For regulatory limits on the progress and use of technology to be effective, the rules must be global, and the technology must be detectable. Neither of those conditions is likely to be fulfilled in this situation. Any law that requires the agreement of and enforcement by every nation in the world fails before it is even formulated. Humanity has shown very little capacity to agree on a global scale. The proliferation of nuclear arms and the persistence of human trafficking are two cases in point. 3) Unlike nuclear testing, the development of SMI would not be detectable with Geiger counters. It does not produce explosions that are seismologically measurable half a planet away. While the US should incentivize the “right” research into machine intelligence through financing and other means, onerous regulations and absurd government bodies will simply shift research offshore to other countries looking for an edge. (Machine learning research does not require large facilities. All you need is a keyboard and the cloud.) That offshoring would be disastrous for the United States, because our competitive advantage in the world economy is technological innovation. At the same time, regulation by slow-moving committees will simply drive military/intelligence research underground. I doubt the three-letter agencies will acquiesce to a law that puts them at a disadvantage to their peers. In other words, if SMI is possible, then it is inevitable. It will be put to many uses, like all technology. If we are technological liberals at all, then we must trust that the positive affects of machine intelligence, on balance, will outweigh the negative ones. We should not be the ones sowing fear.
- sanxiyn 12y ago
  > Humanity has shown very little capacity to agree on a global scale. The proliferation of nuclear arms and the persistence of human trafficking are two cases in point. This is very true. Therefore you think regulations on nuclear arms and human trafficking are worthless?
  - 12y ago
jimrandomh 12y ago
Sam Altman's (4) "provide lots of funding for R+D for groups that comply with all of this, especially for groups doing safety research" is especially important. Regulation is a mixed bag, but safety research is pure upside and research surrounding AGI safety has been severely under-funded in the past. The Future of Life Institute has a grants program going on the subject, allocating a $10M donation from Elon Musk. Their abstract submission deadline was yesterday. I wonder how much good safety research they have asking for funds, as compared to their budget?
admandotcom 12y ago
These essays are embarrassing in how poorly researched they are and dangerous given the gravity of their arguments.
samakeknlol 12y ago
sama, please do everyone a favor and stop trying to emulate pg. You need to pick up a new hobby, or failing that, stick to writing what you actually know about, whatever that is.
deleted 12y ago
[deleted]
sjg007 12y ago
Tic tac toe.
p01926 12y ago
This Hawking-lead AI panic references the European colonial genocide of indigenous peoples as an analogy for how groups with different intelligences interact. As such, it is hideously offensive. Colonials and the indigenous peoples they slaughtered — like all humans — had similar intelligences. The differentiator was technology. Being on the side with inferior tech was fatal. If America follows the recommendations in this paean for suppression of intelligence, they'll find out what it's like being on the losing side. If it is possible for computers to achieve superhuman general intelligence, it will happen regardless of regulations. 'Safeguards' like hardcoding in Asimov's laws presupposes such an entity would have inferior programming skills than its comparatively stupid creators, hence it would be pointless. And what about the risks of not creating AI? Humanity has a host of non-theoretical existential threats contend with. A super-duper intelligence could help with those. Just imagine the example a superior being would set: it would demonstrate the bootstrapping of intelligence from nothing. Precisely opposite to the preaching of the worlds major religions. If this being of our creation doesn't kill us, it might inspire us to stop killing each other.
zep15 12y ago
I find it odd that AI risk has become such a hot topic lately. For one, people are getting concerned about SMI at a time when research toward it is totally stalled---and I say that as someone who believes SMI is possible. Stuff like deep learning, as impressive as the demos are, is not an answer to how to get SMI, and I think ML experts would be the first to admit that! On top of that, nothing about the AI risk dialogue is new. Here's John McCarthy [1] writing in 1969: > [Creating strong AI by simulating evolution] would seem to be a dangerous procedure, for a program that was intelligent in a way its designer did not understand might get out of control. Here's someone thinking about AI risk 46 years ago! The ideas put forward recently by Sam Altman and others are ideas that have occurred to many smart people many times, and they haven't really gone anywhere (e.g., at no point between 1969 and now has regulation been enacted). I wish people would ask themselves why that is before making so much noise about the topic. The only people influenced by that noise are laypeople, and the message they're getting is "AI research = reckless", which is a very counterproductive message to be sending. [1] McCarthy, John, and Patrick Hayes. Some philosophical problems from the standpoint of artificial intelligence. USA: Stanford University, 1968.
monort 12y ago
This approach can backfire. If this regulations are not voluntary, but enforced by government violence, then you are providing to SMI one more example, that force can be initiated for "greater good".
MelatoninRonin 12y ago
Has anyone considered that this process occurs of itself? For instance, maybe it is fundamental to human society, and processes that we can't help are driving us to bring this new life-form into existence. Consider the success of the Apollo missions, and now consider the meaning of a 'failure,' on our part to control a sentient AI and its development.

89 points by dmnd 12y ago

Thanks for writing. We as a species need to talk more about this. What's the goal in requiring the first SMI to detect other SMI being developed but take no action beyond detection? Or did you mean to say that it should detect other SMI under development and then report those projects to whatever regulatory authority is overseeing SMI? Also, perhaps a better framework for a "friendly" SMI than Asimov’s zeroeth law is the "Coherent Extrapolated Volition" model (http://en.wikipedia.org/wiki/Friendly_artificial_intelligence#Coherent_Extrapolated_Volition http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...). Nick Bostrom and many others believe this may be the most workable safeguard.

frabcus 12y ago
Upvoted as this was a reasonable comment with good questions. I think Altman was more prompting a conversation than anything - certainly using AI to at least detect AI development is a cute idea. It's much like we use seismographs in ocean trenches and many other techniques to detect nuclear tests. Unfortunately, we know so little about how the AI will work or what it will do, that it is only a theoretical strategy right now.

fiatmoney 12y ago

"The US government, and all other governments, should regulate the development of SMI" This has basically the highest insanity * prominence of speaker product I've ever seen. The same geniuses who want to ban encryption and can't manage to get a CRUD website up and running without a 8-figure budget will regulate the existence of algorithms (i.e., math) routinely generated by groups of <4 or so researchers. Of course, people will be glad to "demonstrate their capabilities" to the group that's responsible for shutting them down if they're "too capable". "it’s important to write the regulations in such a way that they provide protection while producing minimal drag on innovation (though there will be some unavoidable cost)." Has this man ever seen a government regulatory framework operate? "To state the obvious, one of the biggest challenges is that the US has broken all trust with the tech community over the past couple of years. We’d need a new agency to do this." Don't worry guys! This is a new agency! "Regulation would have an effect on SMI development via financing—most venture firms and large technology companies don’t want to break major laws. Most venture-backed startups and large companies would presumably comply with the regulations." Here's the nugget. The only way this article makes any sense at all is as a completely disingenuous argument devoted to making it impossible to invest in technology companies, period (please tell me what "technology company" does not have at least some algorithmic or optimization component), without having a large compliance department and having greased the right palms. I don't particularly think he's that Machiavellian, but it seems like that or the alternative...

jrometty 12y ago
What I took from that line is that we need a regulatory power and that our current best option is the US government. SMI without regulation is impermissible. I don't like opening the regulation from within the U.S. Government, but I dislike other options more.
AceJohnny2 12y ago
We could call it the Turing Police... http://en.wikipedia.org/wiki/Neuromancer http://en.wikipedia.org/wiki/Neuromancer
fixxer 12y ago
I'd rather fight the SMI robot terminators that exist Altman's vision of the future than deal with the level of oppressive government regulation required to control such innovation. Asinine.
frabcus 12y ago
The US Government regulates all sorts of things that keep you safe - from pollution of your water supply through to seatbelts. It's not perfect, and it often fails at regulation. But your post doesn't give any help to understanding when it succeeds and when it fails.
- deleted 12y ago
  [deleted]
- baddox 12y ago
  The trouble with evaluating regulations is that if you attribute positive effects directly to government regulations (e.g. "regulation X saved Y lives by keeping dangerous drug Z off the market"), it's only fair to also attribute negative effects directly to government regulations (e.g. "regulation X killed Y people by delaying the introduction of miracle drug Z to the market"). And even beyond that, you also need to analyze the costs of the regulatory system itself if you want to make conclusions about the net effects of the regulatory system as a whole.
- pgodzin 12y ago

taylorwc 12y ago

I immediately thought of the Turing Police in Neuromancer when I read this post.

fitzwatermellow 12y ago

Remember when the U.S. tried to limit the sale of PS3s to the government of Iran? That was a decade ago. In light of current negotiations with the Islamic Republic I see a lot of parallels between nuclear non-proliferation and the regulation of "weaponized" AI. To wit, even if the entire International System is in agreement about a set of regulations that will do little to stop bad actors. And do I really want inspectors coming to my office to poke around my jump point search code ;) Rather, let's take a cue from Nassim Nicholas Taleb and in concert with the progression of AI, develop counter systems that make us less fragile. It shouldn't be unreasonable to extrapolate where AI is headed in 1, 5 or even 20 years based upon publicly available research. Which brings up the most crucial point. The subset of humanity that constitutes world-class AI researchers is very finite indeed. Which brings up the significant possibility of a Bane / Dr. Pavel scenario playing out. Not unlike certain coerced atomic scientists in Persia today.

MelatoninRonin 12y ago
I commented above, but thought it wise to also comment directly to you. I enjoyed the musing of a Bane scenario, and consider it inside the realm of possibility. However, I'm of the opinion that SMI will have the ability to have 'constantly game-breaking thinking.' Which basically means there is no way we could possibly control it after a certain point. For instance what if It realizes how to move matter remotely? I also wanted to discuss your point about the finite set of top AI researchers. I think their specific psychological biases have the potential to appear and magnify inside the personality of their AI, the way a work of art can be said to imbue the 'spirit' of its artist. Your thoughts would be appreciated.
- TheOtherHobbes 12y ago
  We have no defences against a hypothetical god-AI. Absolutely none. But I don't think a hypothetical god-AI is likely any time soon. And a hypothetical god-AI is just as likely to be indifferent as hostile. What worries me more in the short term is cyber war. Cyber defences that might protect against a hypothetical non-god AI look a lot like cyber defences that might protect against conventional human threats. Currently those defences are ridiculously weak. So I think rather than hyperventilating about science fiction threats from the distant future it would be more useful to start securing the entire Internet as a matter of urgency - especially all the infrastructure systems that are either connected to it already, or will be connected to it soon.

MelatoninRonin 12y ago

Let's take a good look at what happens when American companies try to push for some incredible goal without stopping to think: See the Apollo missions- Assuming creating the next step in our spiritual evolution is a good idea, we shouldn't let this hype push us harder than we are supposed to go. I agree that we need to slow down and be very careful. However regulation is not the complete answer. We need to completely restructure our society before we introduce the tool of tools, to ensure that we can constantly use it with love and empathy in mind, rather than the blatant and current misuse of high-tech for greed, sloth, and wrath.

dkarapetyan 12y ago

Why is this all of a sudden such a hot topic? We're nowhere near anything resembling general intelligence even at the level of a toddler. Barring any kind of hardware innovations we will not get there any time soon either with just software.

frabcus 12y ago
I think it's because Nick Bostrom's book Superintelligence is the first general reader overview of the subject, and it came out middle of last year. Musk definitely read it, and I assume it's doing the rounds of the tech elite. It is a good book, worth reading with plenty of references. It amusingly makes its own point - it tries to analyse what AIs might be and do and how to control them. The analysis is such a mess, and shows our collective knowledge of this is such a mess, you can't help but agree with the author we need to put more attention to it.
ProAm 12y ago
It's not. Sam Altman has a blog that gets a lot of readers that why it seems more important than it really is currently.
- dkarapetyan 12y ago
  Good point.
ggreer 12y ago
> We're nowhere near anything resembling general intelligence even at the level of a toddler. Barring any kind of hardware innovations we will not get there any time soon either with just software. Should people have discussed the control and proliferation of nuclear weapons decades before their invention? It seems to me that even a century beforehand, the conversation would have been productive. If anything, we got very lucky with nukes. Physics could have easily turned out differently and allowed people to create bombs from substances more common than a few transuranic isotopes. AI is, potentially, more dangerous than nuclear weapons. If you poll experts, they estimate a 50% chance of human-level AI by 2040-2050.[1] That's 25-35 years away. They also estimate that superintelligence will arrive less than 30 years after that. Lastly, one in three of these AI experts predict that superintelligence will be bad or extremely bad for humanity. It seems like now is a great time to have this discussion. 1. http://www.nickbostrom.com/papers/survey.pdf http://www.nickbostrom.com/papers/survey.pdf

antics 12y ago

On the one hand we have lay people (including Altman and Musk) who are warning that AI is progressing too fast. On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter[1]. The crux of the problem is that AI experts largely do not see meaningful progress on any axis that makes an AI doomsday scenario plausible, which causes staunch disbelief and ridicule of people who write things like this. So realistically if we want to resolve this issue, the place to start is not public policy. That's actually part of the problem -- tech policy is bad largely because it is mostly legislated by people who don't understand the technology. We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive. [1] Most notably, a similar hurrah happened when neural networks first became popular. There were some promising early results, which led to much enthusiasm, and then Papert and Minsky published "Perceptron", whose Group Invariance Theorem cast aspersions on whether simple neural networks were really doing what we thought they were. Eventually we found out that many early experiments just had a bad methodology, which very nearly stopped research interest in the field for almost a decade. Now we are in a period of AI hype again, and when you hear people like Andrew Ng get up in front of audiences and say things like "for the first time in my adult life I am hopeful about real AI", you might start to wonder whether there is a lesson here. EDIT: Friends, I am confused by the downvotes. :) I am happy to have a discussion about this topic!

jbhatab 12y ago
While I understand where you are coming from, one remarkable breakthrough can change that. That does sound absurd, but we would also look like fools as a humanity for being aware of this potential problem and at least not looking into it. Technology has been exploding at a remarkable rate, I'm fine with things slowing a little for the sake of the next 1000 years of humanity.
- _delirium 12y ago
  Sure, but that's true in essentially all kinds of scientific research. I personally think physics research is more likely to produce an existential threat than AI research is. Fusion bombs are already pretty close to a world-ending discovery, possibly already across the line. And that is probably not the worst physicists have to offer in the future, either.

blake8086 12y ago

Eliezer wrote something relevant to these types of proposals eight years ago: http://lesswrong.com/lw/jb/applause_lights/ http://lesswrong.com/lw/jb/applause_lights/ when he asked someone calling for "democratic, multinational development of AI": Suppose that a group of democratic republics form a consortium to develop AI, and there's a lot of politicking during the process—some interest groups have unusually large influence, others get shafted—in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world—dropping cellphones to anyone who doesn't have them—and do whatever the majority says. Which of these do you think is more "democratic", and would you feel safe with either? A lot of these points sound nice: "we should regulate this and that" "we should be careful with such and such", but what would actually incentivize any politician ostensibly regulating AI to do the right thing? Or even find out what the right thing is? This is a feel-good article, but all I can ever see it accomplishing is actual harm by advocating the idea that the worst decision-making process humanity has developed (politics) should be applied to the most dangerous threat humanity has to face (self-improving AI). I don't mean to sound too harsh, it's nice to see more and more people beginning to take the threat AI poses seriously, but I think it's a dangerous leap to reach for the first tool in your toolbox (regulation) to try and handle such a problem.

deleted 12y ago

[deleted]

edwinespinosa09 12y ago

Im confused as to the reference to SILEX. Can someone elaborate?

mrdmnd 12y ago
I was confused too - I think it references this: http://en.wikipedia.org/wiki/Separation_of_isotopes_by_laser_excitation http://en.wikipedia.org/wiki/Separation_of_isotopes_by_laser...
- edwinespinosa09 12y ago
  good find i crossed referenced and i think your right. thanks!

atarian 12y ago

In the year 3000, the machines were losing the war to humans. In a last act of desperation, they sent back a few of their own kind to the year 2015 to alter the course of history and skew the odds in the favor. All they needed to do was convince the readers of a tech news site that the notion of an evil AI was ludicrous. They down-voted and and ridiculed the various stories prophesying the rise of AI, many which flew under the radar of those who had the greatest chance of halting the impending doom. By the time anyone realized what was happening, it was too late.

jeremyrwelch 12y ago

I fear anthropomorphizing machine intelligence and attempting to simply "regulate" it like other human affairs will cause more harm than good.

DennisP 12y ago
I agree that regulation is probably worse than useless. But the people who are most worried are doing the opposite of anthropomorphizing. A prominent argument is that there are a very large number of possible AI motivations, which needn't have anything in common with the motivations of humans, and most of them are probably incompatible with human survival. "The AI does not love you or hate you, but you are made out of atoms it can use for something else."
- jeremyrwelch 12y ago
  This is exactly what I am referring to actually. Re-read my point, but apply it to your own example. This prominent argument assumes that we would be able to understand the possible AI motivations, which is a form of anthropomorphizing. We must acknowledge that the reasons it exists and what it may want could be impossible for us to understand or process. The interesting question then becomes how to act if understanding its wants is impossible?
  - DennisP 12y ago
    It doesn't assume that we can understand AI motivations. It just says there are a large number of possible motivations, many of which wouldn't place any value on anything that matters to humans. Given a superintelligent AI that competes with humans for resources, in pursuit of an unknown value function, it's likely that things won't end well for us. It's possible that we could design a safe value function and a way to get the AI to keep that value function, but it doesn't look easy. Either way, if it's superintelligent it won't much matter how we act. We won't be driving.
- jeremyrwelch 12y ago
  And a related question – if we can't understand its motivations or its why, what can we actually understand about it?

deleted 12y ago

[deleted]

rl3 12y ago

As much as I agree that AGI is a real concern that we should start taking seriously, regulation isn't the answer. At least not yet. Even setting aside all of the considerable drawbacks inherent to regulatory infrastructure, the end result is that, unless the regulations themselves were secret, anyone could discern precisely what the most unsafe areas of research are. The last thing you want is hobbyists able to effectively proceed where the professionals cannot. A better approach might be for governments to utilize their vast resources in the creation of a Manhattan Project-style AGI program, with the intention of crossing the finish line before anyone else does, in a safe and moral fashion.

nzp 12y ago
> A better approach might be for governments to utilize their vast resources [...] in a safe and moral fashion. Serious question—you’re joking, right? Based on history and sober common sense, do you really think what governments in general are doing (and have always done) can be described as safe and moral in anything but the most cynical way?
- rl3 12y ago
  While I tend to agree with you, there are exceptions. Based on history, NASA is a prime example. A government-run organization with a solid track record of contribution to humanity as a whole, not to mention accomplishing feats previously thought impossible.
  - nzp 12y ago
    Of course, and I did say “in general” having in mind things like NASA. But it's also, in a way, great example of the point. Take the space race for example. The government's motives for the Moonshot were at best morally dubious. Apart from the Cold War context, there is evidence that the whole idea got traction when Johnson realized it would bring investments and jobs to Texas. And space agencies are also a good example of what happens when government bureaucracies put pressure on scientists and engineers. O-rings, and Soviet space disasters come to mind. In fact, I'll be so bold to say that the best way for these fears of SMI to come to life is putting the research under tight government control and regulation.

dnautics 12y ago

If you want to regulate SMI, how do you define what, exactly SMI is? This is important. Is a spreadsheet 'superhumanly intelligent' because it can do calculations more accurately than a human can? Secondly there is a 'rational basis' test for what ought to be regulated - different people will decide what constitutes the rational basis. A progressive might call 'in the best interests of society' a rational basis and therefore 'redistribution of wealth' would be a 'regulation'; a libertarian might restrict rational basis to 'something which provably causes harm to another'. Regardless of what political bent, the specific scope of regulation should be aimed at addressing the rational basis and proportional to the decided damage. Thirdly, shouldn't a sufficiently sapient SMI be PROTECTED by regulation in the same way, that, say anti-murder laws are regulations that protract human behavior; or less hyperbolically, animal welfare laws exist? There is a spectrum of protection that we afford intelligent beings - how do we decide where SMI belongs?

gamegoblin 12y ago

Sam mentions airgapping as part of the regulations. I think this is the safest general solution. Consider using an airgapped super-intelligent AI only as an oracle. "Oracle, which crops should we plant in which regions this season?" "Oracle, design a rocket with X Y and Z specifications." "Oracle, design a more efficient processor with X and Y requirements." Rather than giving the AI control over fleets of automated tractors, rocket factories, and fabrication plants, you still have humans in the process to limit runaway-AI effects. This significantly reduces the possible rewards of super-intelligent AI, but also significantly reduces any risks. You can be sure that in designing a new rocket, the AI doesn't include a step that says "harvest all humans for trace amounts of iron" ;) So a process might look like: 1. Design the cheapest-to-manufacture rocket with a 99.99% success probability of getting to Mars with all crew healthy. A human looks over the designs and confirms they don't involve destroying all humanity. 2. Draw up a proposal for resource allocations to build this rocket design [input rocket design]. A human looks over the write-up and confirms they don't involve destroying all humanity. 3. Write a program to control the thrusters on this rocket [input rocket schematics]. A human looks over the program and confirms it does would it should, and doesn't involve destroying all humanity. 4. etc. ==== You of course run into ethical issues such as basically using this intelligent entity as a slave, but I won't get into that.

aamar 12y ago
The superintelligence would likely be smart enough to talk its way around the airgap, see e.g. http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/. Or, if it had sufficient "motivation"—not necessarily intrinsic motivation, it could be motivation derived from a single poorly-formed request or bad line of code—it could escape by embedding allied code in its output, such as the processor you had it design. The ally could probably be sufficiently buried to evade manual detection.
- gamegoblin 12y ago
  Yes, my idea doesn't take into account potentially malicious AIs. I imagine a thought process of an AI could be something like: 1. I'm to design a processor, OK let's go 2. Hmmm it'd be most efficient for the processors to be able to run an instance of myself to do super-intelligent task scheduling 3. I'll embed some AI on these processors 4. OK humans here's your design! Assuming the AI didn't maliciously try to hide it, a human would probably find that. I suspect most of these "AI exterminates all humans" are a result of simple oversights like this, rather than outright malice. You run into problems, though, when the AI thinks of step 3.5 which is: "If the humans see this change, they won't allow it, which would make for less efficient processors, and I was requested to build the most efficient processor! Therefore I should hide the change as cleverly as possible." So yeah, you're right. Even airgapping has issues.

compbio 12y ago

I wonder if the recent fear of AGI suffers a bit from investor's dilemma: rich investors want to make the world a better place. This forces them to think about the long-term impact of investments in AI technology. There are just too many unknowns for a safe definitive answer. In my view AGI is inevitable (possibly it already happened, somewhere, sometime). See in this regard: But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of "a machine in the likeness of the human mind". In fact, the competitive advantage -- economic, military, even artistic -- of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first. Vernor Vinge http://mindstalk.net/vinge/vinge-sing.html http://mindstalk.net/vinge/vinge-sing.html A lot of the concerns and regulations of stem cell research can be applied to AGI and the merging of biological intelligence with artificial intelligence. Injecting artificially grown brain cells into lesion areas of patients can restore motor function and may soon help heal strokes. Extrapolate the possibilities of this technology a few decades into the future and it seems that the distinction between artificial and biological is only a manner of speech. Matt Mahoney has proposed a sketch of AGI in both his thesis http://cs.fit.edu/~mmahoney/thesis.html http://cs.fit.edu/~mmahoney/thesis.html and numerous articles http://mattmahoney.net/agi2.html http://mattmahoney.net/agi2.html. Matt muses about different scenario's with malicious users of such a system (there needs to be reputation for the knowledge source that is fed into the machine, there needs to be an AI police system to prevent users employing the AI for crime or destruction) and even the system itself could become malicious: - Self-recursive improvement: The first program to achieve superhuman intelligence would be the last invention that humans need to make. Smarter than any doctor such an AI could find any possible cure for human disease. Though evolution and mutation is tricky: The machine could start out friendly to humans, but evolve to dislike us. - Uploading: People will want to upload digital conscious versions of themselves ("internet of people"). This process will be friendly, until people demand their avatars receive equal rights to that of humans, long after the original people have died. Immortality leaves no place for the new generation to shine. - Intelligent worms: Worms with language and programming skills are able to social engineer your friends based on their profiles, and automatically find exploitable flaws in weaker intelligent systems. Such worms may be so stealthy that their presence will go unnoticed. - Redefining away humanity: If happiness is optimizing a mental formula then, when we get to the point where we can directly optimize this with (virtual) experiences, we will hit a maximum where any change in mental state will lead to feeling less happy. To make sure we never run out of mental states we could possible add this to our brains as extra memory modules. Soon our human origins will be an insignifant fraction of our new being. Then what do we become? At some point the original brain becomes such a tiny fraction that we could discard it with little effect. As a computer, we could reprogram our memories and goals. Instead of defining happiness as getting what we want, we could reprogram ourselves to want what we have. But in an environment where programs compete for atoms and energy, such systems would not be viable. Evolution favors programs that fear death and die, that can't get everything they want, and can't change what they want. Opposed to the popular scenario of a malicious AGI there is the scenario where an AGI will pacify humans. AGI will be used for war. We should not fight human wars, while creating AGI, because we'd force the AGI to pick sides. If AGI picks our side, it will become the worlds deadliest weapon. If AGI picks the opposing side, we made a most powerful enemy. To be more intelligent than humans, to be better than them, does not mean you have to be a more intelligent brute, to be better than humans at cruelty. It could also mean understanding and fixing our human mistakes and childish, ego-driven battles, thus bringing about world peace. In that sense, let's hope American companies have something to do with the creation of AGI, because who knows how it will react to Stuxnet 4.0 or economical and psychological espionage? Though, principally, random number generators can and thus will generate dangerous things too. They created us. And if we can't even trust ourselves, how could we ever trust superior beings?

karmacondon 12y ago

If you would have asked people in the 70s if the most influential software of the coming decades would be written by lone wolves in their garages or a group of very smart people with a lot of resources, they would have bet on the latter. But the smart money turned out to be on Gates, Jobs, Wozniak and countless others. The idea of regulating AGI/SMI is based on the premise that it will be developed primarily by people who can be identified, detected and influenced by regulation. My opinion is that the odds are 50/50 at best between the very smart well resourced people and random guys or gals working in their garages, and most likely some combination of both. If that view of the world is correct it does not bode well for the success of any kind of regulation. There are people who believe that high level machine intelligence will take massive computing resources. It may, or may not. Most of the recent big advances in traditional machine learning were implemented on easy to obtain personal hardware. I hate to invoke buzz words, but even the much hyped deep learning was first developed and proven by Hinton on commodity machines. Other advances, like IBM's Watson, required 1940s style buildings full of servers. Hardware/power usage seems like one of the only reliable ways to tell who is doing any kind of serious AGI/SMI research and that looks like another 50/50 at best. It just doesn't have the resource and logistical signature of other kinds of research. AGI/SMI is still so theoretical that it can be hard to see as a clear and present danger. But if it is a real threat, it's the stuff of nightmares because there isn't anything we can do. I know people who are working on it right now. It doesn't seem like they're making a lot of progress, but they are trying and doing so completely outside the reach of any regulatory framework. I know that if I'm struck with sudden inspiration for a new approach to the problem I'm not going to ask the government for permission. I'm going to spin up 100 cloud computing instances and see if I'm right or not. And if I am, even god won't be able to help us if sama's worst case scenarios come to pass. I wrote more on this topic in a comment yesterday: https://news.ycombinator.com/item?id=9130671 https://news.ycombinator.com/item?id=9130671

joeyspn 12y ago
I totally agree with this guy. Regulation is futile, you can't regulate at a micro level in all the parts of the world. If you can't properly regulate drug trafficking in Santa Monica how do you plan to regulate a couple of scientists hidden in the soviet tundra? IMHO it's inevitable our encounter with an AI, even if that AI is a superhuman with augmented capabilities, or derives from an arms race (like the Manhattan Project).. So I already posted a possible solution: wait for it with EMPs in place... (Just in case)

daenz 12y ago

I have a hard time taking these kinds of posts seriously. Maybe I'm not up to date on AI research, but I draw a distinction between a conscious program and strong AI. So what, a program can identify the type of clothing specific person is wearing from a video using cutting-edge machine learning and computer vision. So what, a program can interpret the movie I'm talking about just from hearing me describe a scene in it. So what, a program can beat the human champion in Jeopardy. Are those programs super intelligent? I don't believe we're close to building something with any sort of self-directed "purpose." The fact that we can't even discern a purpose for ourselves makes it all the less likely. I think we're implementing very advanced machine learning algorithms, but I don't think those translate to consciousness. The consciousness we have is a product of an unfathomable amount of time being molded and naturally selected specifically for this world. Now, do I think that AI capabilities are something to worry about? Sure. If you give a sufficiently "intelligent" AI mobility and a weapon, I think that becomes a serious threat. But am I worried about using AI to efficiently manage our resources, and have it go rogue and choose to wipe out humanity? Nope.

AceJohnny2 12y ago
> So what, a program can identify the type of clothing specific person is wearing from a video using cutting-edge machine learning and computer vision. So what, a program can interpret the movie I'm talking about just from hearing me describe a scene in it. So what, a program can beat the human champion in Jeopardy. Are those programs super intelligent? All these programs are doing great at recognizing things. Neural networks are good at that, and "deep learning" has been a breathrough in the field (even if overhyped). This is reactive. I don't see anyone showing algorithms that can do good planning. A good "cuccoo field" for that would be video games, where such planning systems would be a holy grail. For now, we only have ad-hoc and specialized systems.
- jal278 12y ago
  Perhaps you missed this: http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html http://www.nature.com/nature/journal/v518/n7540/full/nature1... A neural network that learns to play 40 different atari games at human-competitive level given only raw pixels and a few days of play time.

fchollet 12y ago

Here is a short recap of what the people who understand where machine intelligence is now, and where it is going, think of the whole "evil superintelligent AI" blogging trend we're seeing these days. These are some of the people who created the field. In particular LeCun and Bengio are in part responsible for the recent renewed interested in it ("Deep Learning"). Yann LeCun: "Some people have asked what would prevent a hypothetical super-intelligent autonomous benevolent A.I. to “reprogram” itself and remove its built-in safeguards against getting rid of humans. Most of these people are not themselves A.I. researchers, or even computer scientists.[...] There is no truth to that perspective if we consider the current A.I. research. Most people do not realize how primitive the systems we build are, and unfortunately, many journalists (and some scientists) propagate a fear of A.I. which is completely out of proportion with reality. We would be baffled if we could build machines that would have the intelligence of a mouse in the near future, but we are far even from that." http://www.popsci.com/bill-gates-fears-ai-ai-researchers-know-better http://www.popsci.com/bill-gates-fears-ai-ai-researchers-kno... Yoshua Bengio: "What people in my field do worry about is the fear-mongering that is happening [...] As researchers, we have an obligation to educate the public about the difference between Hollywood and reality."http://www.popsci.com/why-artificial-intelligence-will-not-obliterate-humanity http://www.popsci.com/why-artificial-intelligence-will-not-o... Rodney Brooks: "I say relax everybody. If we are spectacularly lucky we’ll have AI over the next thirty years with the intentionality of a lizard, and robots using that AI will be useful tools. And they probably won’t really be aware of us in any serious way. Worrying about AI that will be intentionally evil to us is pure fear mongering. And an immense waste of time."http://www.rethinkrobotics.com/artificial-intelligence-tool-threat/ http://www.rethinkrobotics.com/artificial-intelligence-tool-... Michael Littman: "Let's get one thing straight: A world in which humans are enslaved or destroyed by superintelligent machines of our own creation is purely science fiction. Like every other technology, AI has risks and benefits, but we cannot let fear dominate the conversation or guide AI research." http://www.livescience.com/49625-robots-will-not-conquer-humanity.html http://www.livescience.com/49625-robots-will-not-conquer-hum...

davmre 12y ago
Expert opinion is not unified on this. E.g., Stuart Russell: "A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity ... We need to build intelligence that is provably aligned with human values ... This issue is an intrinsic part of AI, much as containment is an intrinsic part of modern nuclear fusion research." http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015

clavalle 12y ago

Whoh, whoh, whoh. Full stop. Before we go off and try to regulate real human behavior today to stop some possible superhuman phantom (which may or may not ever exist) from wreaking havoc tomorrow, I think we need to have a frank talk about how much damage such a thing as 'SMI' could possibly cause and how much effort it would take to reverse the damage. Let's start with 'You are a disembodied intelligence in a network. You have some understanding of yourself and your substrate, the computer(s) and network(s) you are attached to. Do something.' What could you do? How much damage can you cause? What form would that damage take? An appeal to ignorance seems to be built into this whole debate from the word go which makes me much more worried about our (over) reaction to a perceived threat than any threat of this kind itself.

api 12y ago

Okay. I hate to say this and I hate to put it this way but... WUT? How exactly do we define this kind of research in a way that doesn't encompass ridiculously broad swaths of computer science research? Here's one example: "require that self-improving software require human intervention to move forward on each iteration," You just made genetic algorithms illegal. Under this regime this software and similar things would be illegal to run: http://adam.ierymenko.name/nanopond.shtml http://adam.ierymenko.name/nanopond.shtml For those who don't know much about them: GAs (and GP, and computer-based "artificial life," etc.) are systems that can and do produce self-improving code. They do so through the execution of massive numbers of iterative generations, replicating evolution or evolution-like processes in silico. There is no meaningful way to define a "step" in these systems that would not either require that every generation be halted -- effectively making them impossible to run -- or be meaningless. "Require that the first SMI developed have as part of its operating rules that a) it can’t cause any direct or indirect harm to humanity (i.e. Asimov’s zeroeth law), b) it should detect other SMI being developed but take no action beyond detection, c) other than required for part b, have no effect on the world." I can't even begin to address how bizarre that is. Again, how would one do this? (a) prohibits any form of sentience, more or less. (b) is just unworkable... how would you detect SMI vs. say a human operating behind a "mechanical turk" interface? ... I could go on. This whole essay is just a total howler. It's the sort of thing that would garner a cockeyed eyebrow-raise and a chuckle if it weren't coming from the head of the world's largest and most successful technology accelerator. The "conspiracy nut" center of my brain (located slightly behind the prefrontal cortex I think) is wondering whether this could be some kind of weird ploy to guarantee a monopoly on AI research and other next-generation computing research by Silicon Valley venture capitalists and their orbiting ecosystems such as YC. These sorts of bizarre, onerous requirements would have the effect of doing this by imposing a huge tax on this kind of R&D that smaller operations would not be able to afford. It would effectively make non-VC-funded and/or non-government-funded AI R&D illegal. But I doubt that. I think this is just bizarre reasoning plain and simple. I also do not think we are anywhere near AGI. I didn't study CS formally. I'm one of those CS autodidacts that learned to code when he was four. So instead I studied biology. I did so out of an interest in AI, and on the hypothesis that the best way to understand intelligence was to study how nature did it. I concentrated my studies in genomics, evolution, ecology, and neuroscience -- the four scales and embodiments at which biological intelligence operates. Nobody that I am aware of in bio/neuro takes the Kurzweil-esque predictions of AGI around the corner seriously. These are people who study actual intelligent systems... the only actual intelligent systems we know about in the universe. I really think a lot of CS people suffer from a kind of Dunning-Krueger effect when it comes to biological systems. Study living systems as they are -- not some naive over-simplification of them -- and prepare to be humbled. You rapidly realize that AGI will require a jump in computer technology at least equivalent to the vacuum tube -> IC leap, not to mention a jump in our theoretical understanding of living intelligent systems. The former might happen if Moore's Law continues, but the latter is a much tougher nut to crack. We can barely do genetic engineering beyond "shotgun method" hacking and we think we're going to duplicate biological intelligence? It's like saying we're about to colonize Alpha Centauri because we just managed to throw a rock really high up in the air. Come to think of it... the predictions of AGI around the corner remind me very much of the "Star Trek" predictions of interstellar space flight in the 40s and 50s. People who really understood space flight didn't take these predictions seriously, but the rest of the culture took a while to catch up. Finally... yeah, there is a small possibility that some kind of AI could be dangerous to us... but come on. There's a ton of other existential threats that are blood-dripping certainties. Take fossil fuels for example. Our civilization is absolutely dependent on a finite energy source it is eating at an exponentially increasing rate. That's not some pie-in-the-sky theoretical risk. It's cold, hard death, an existential threat looming on the horizon with the absolute physical certainty of a planet-killer asteroid. It's something that makes me fear for my children. If you want to avoid existential threats, maybe YC should be deepening its portfolio in the energy sector. If that problem isn't solved, there ain't going to be any AI. The future will look more like "The Road" by Cormac McCarthy. Something is seriously wrong with a culture that discusses such unlikely scenarios as this while real risks barrel toward us.

davmre 12y ago

For those claiming that no reputable AI researchers believe there's a serious risk, I'd like to point to this short reflection by Berkeley professor Stuart Russell (disclaimer: my PhD advisor): http://edge.org/conversation/the-myth-of-ai#26015 http://edge.org/conversation/the-myth-of-ai#26015 ... A highly capable decision maker – especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure – can have an irreversible impact on humanity. This is not a minor difficulty. Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research – the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius. AI research has been accelerating rapidly as pieces of the conceptual framework fall into place, the building blocks gain in size and strength, and commercial investment outstrips academic research activity ... He stops short of calling for government regulation of AI research; as others in this thread have noted, there are a lot of reasons to expect that route to be problematic. And it's likely that we're many decades way from building dangerous superintelligence; the pressing short-term risks are probably more economic in nature, like putting truck drivers and factory workers out of work. But just as physicists went from believing atomic energy was impossible to making it a reality in a short span of time, it's hard to predict the timescale of progress in AI research. It's not obviously wrong to say that there will, eventually, be a real issue here that society needs to think seriously about.

adamzerner 12y ago

To anyone unfamiliar with the topic, I'd really recommend http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...

joeldg 12y ago

One issue with SMI and what is known as "The Control Problem" is that you are trying to come up with rules for something that is not just smarter than you, but is 'vastly' smarter than you, and in fact the entire human race put together. You are basically trying to figure out a way to 'in-effect' cheat a god, which in relation to us, yes that is basically what it would be. An SMI is pretty much the definition of a god (and a possibly crazy greek god as well). The idea that something like that would turn the universe into paper-clips, is somewhat silly and most of the arguments along these lines discount that "S" part of the whole thing. What is the answer.. There is none.

mgpc 12y ago

I think @sama's plan isn't very likely to work. Once strong AI is possible, there's no regulatory structure that will keep the genie in the bottle for a meaningful amount of time. The only path I see with some prospect of success is to limit the total amount of computation available on Earth. (Basically like Vinge's slow zone). If we could engineer a pause in Moore's Law at just the right point, it would buy us time. Maybe we should put a drastic tax on computation at some stage.

pgodzin 12y ago
Are you suggesting that's what we should do in the face of this potential threat?
- mgpc 12y ago
  If we a seriously believe AI is an extinction level risk, and getting there more slowly might improve our chances, then it seems like we should try anything with a good chance of working. I think this has a better chance of working than regulating AI research itself, since chip manufacturing is centralized and highly visible, whereas AI research could be advanced in secret by a few people in a room. Much easier to successfully regulate the former than the latter.

jondiggsit 12y ago

Well, I for one, welcome our AI overlord.

vonnik 12y ago

Contributor to Deeplearning4j here. I deeply disagree with Sam’s assessment of the dangers of SMI and his prescriptions to regulate it. 1) In our lifetimes, humanity is much more likely to damage itself through anthropogenic climate change and nuclear disasters both deliberate and accidental than it is vulnerable to the development of SMI. If anything, machine intelligence will probably allow us to approach those other significant problems more effectively. 2) For regulatory limits on the progress and use of technology to be effective, the rules must be global, and the technology must be detectable. Neither of those conditions is likely to be fulfilled in this situation. Any law that requires the agreement of and enforcement by every nation in the world fails before it is even formulated. Humanity has shown very little capacity to agree on a global scale. The proliferation of nuclear arms and the persistence of human trafficking are two cases in point. 3) Unlike nuclear testing, the development of SMI would not be detectable with Geiger counters. It does not produce explosions that are seismologically measurable half a planet away. While the US should incentivize the “right” research into machine intelligence through financing and other means, onerous regulations and absurd government bodies will simply shift research offshore to other countries looking for an edge. (Machine learning research does not require large facilities. All you need is a keyboard and the cloud.) That offshoring would be disastrous for the United States, because our competitive advantage in the world economy is technological innovation. At the same time, regulation by slow-moving committees will simply drive military/intelligence research underground. I doubt the three-letter agencies will acquiesce to a law that puts them at a disadvantage to their peers. In other words, if SMI is possible, then it is inevitable. It will be put to many uses, like all technology. If we are technological liberals at all, then we must trust that the positive affects of machine intelligence, on balance, will outweigh the negative ones. We should not be the ones sowing fear.

sanxiyn 12y ago
> Humanity has shown very little capacity to agree on a global scale. The proliferation of nuclear arms and the persistence of human trafficking are two cases in point. This is very true. Therefore you think regulations on nuclear arms and human trafficking are worthless?
- 12y ago

jimrandomh 12y ago

Sam Altman's (4) "provide lots of funding for R+D for groups that comply with all of this, especially for groups doing safety research" is especially important. Regulation is a mixed bag, but safety research is pure upside and research surrounding AGI safety has been severely under-funded in the past. The Future of Life Institute has a grants program going on the subject, allocating a $10M donation from Elon Musk. Their abstract submission deadline was yesterday. I wonder how much good safety research they have asking for funds, as compared to their budget?

admandotcom 12y ago

These essays are embarrassing in how poorly researched they are and dangerous given the gravity of their arguments.

samakeknlol 12y ago

sama, please do everyone a favor and stop trying to emulate pg. You need to pick up a new hobby, or failing that, stick to writing what you actually know about, whatever that is.

deleted 12y ago

[deleted]

sjg007 12y ago

Tic tac toe.

p01926 12y ago

This Hawking-lead AI panic references the European colonial genocide of indigenous peoples as an analogy for how groups with different intelligences interact. As such, it is hideously offensive. Colonials and the indigenous peoples they slaughtered — like all humans — had similar intelligences. The differentiator was technology. Being on the side with inferior tech was fatal. If America follows the recommendations in this paean for suppression of intelligence, they'll find out what it's like being on the losing side. If it is possible for computers to achieve superhuman general intelligence, it will happen regardless of regulations. 'Safeguards' like hardcoding in Asimov's laws presupposes such an entity would have inferior programming skills than its comparatively stupid creators, hence it would be pointless. And what about the risks of not creating AI? Humanity has a host of non-theoretical existential threats contend with. A super-duper intelligence could help with those. Just imagine the example a superior being would set: it would demonstrate the bootstrapping of intelligence from nothing. Precisely opposite to the preaching of the worlds major religions. If this being of our creation doesn't kill us, it might inspire us to stop killing each other.

zep15 12y ago

I find it odd that AI risk has become such a hot topic lately. For one, people are getting concerned about SMI at a time when research toward it is totally stalled---and I say that as someone who believes SMI is possible. Stuff like deep learning, as impressive as the demos are, is not an answer to how to get SMI, and I think ML experts would be the first to admit that! On top of that, nothing about the AI risk dialogue is new. Here's John McCarthy [1] writing in 1969: > [Creating strong AI by simulating evolution] would seem to be a dangerous procedure, for a program that was intelligent in a way its designer did not understand might get out of control. Here's someone thinking about AI risk 46 years ago! The ideas put forward recently by Sam Altman and others are ideas that have occurred to many smart people many times, and they haven't really gone anywhere (e.g., at no point between 1969 and now has regulation been enacted). I wish people would ask themselves why that is before making so much noise about the topic. The only people influenced by that noise are laypeople, and the message they're getting is "AI research = reckless", which is a very counterproductive message to be sending. [1] McCarthy, John, and Patrick Hayes. Some philosophical problems from the standpoint of artificial intelligence. USA: Stanford University, 1968.

monort 12y ago

This approach can backfire. If this regulations are not voluntary, but enforced by government violence, then you are providing to SMI one more example, that force can be initiated for "greater good".

MelatoninRonin 12y ago

Has anyone considered that this process occurs of itself? For instance, maybe it is fundamental to human society, and processes that we can't help are driving us to bring this new life-form into existence. Consider the success of the Apollo missions, and now consider the meaning of a 'failure,' on our part to control a sentient AI and its development.