13 ms·

Elon Musk donates $10M to keep AI beneficial

127 points by cocoflunchy 12y ago

FatalLogic 12y ago
I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity? Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though. Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.
- blfr 12y ago
  It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.
  - FatalLogic 12y ago
    Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
    - ThrustVectoring 12y ago
      Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
    - JamesArgo 12y ago
      It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
      - TeMPOraL 12y ago
        The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
    - eli_gottlieb 12y ago
      > Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
  - rl3 12y ago
    > Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
- pavel_lishin 12y ago
  I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI. The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
  - ceejayoz 12y ago
    > how do you trap and imprison something smarter than you? Jails seem to have this figured out just fine.
    - JamesArgo 12y ago
      The correct analogy would be a jail run by a bacterium.
      - Retric 12y ago
        Prison bars are dumber than a bacterium and they work just fine. Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
        TeMPOraL 12y ago
        The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.
        jacquesm 12y ago
        This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.
- mentos 12y ago
  What about limiting AI to a certain clock rate?
  - kanzure 12y ago
    > What about limiting AI to a certain clock rate? Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
  - FatalLogic 12y ago
    Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.
  - JulianMorrison 12y ago
    http://www.rifters.com/real/2009/01/iterating-towards-bethlehem.html http://www.rifters.com/real/2009/01/iterating-towards-bethle... It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
  - rl3 12y ago
    I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution. For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
- gradys 12y ago
  If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?
  - eli_gottlieb 12y ago
    Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.
    - TeMPOraL 12y ago
      It probably wouldn't need bitcoin; it could figure out ways to trick humans into paying for the servers or just hack its way into electronic transfers.
- Synthpixel 12y ago
  The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies. I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry. Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
  - butwhy 12y ago
    " The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.
- yoha 12y ago
  About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment https://en.wikipedia.org/wiki/AI_box_experiment
- evanspa 12y ago
  "If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual." They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
  - pzxc 12y ago
    Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended. God, I'm such a nerd. :P
    - colordrops 12y ago
      So the holodeck computer must be insanely intelligent. Or, it turns out that the building blocks for intelligence are simple and deterministic.
    - deleted 12y ago
      [deleted]
    - evanspa 12y ago
      Yup. Good catch :)
  - comrade1 12y ago
    He doesn't sound that smart...
- Retra 12y ago
  Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists? We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions. This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish. So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous. Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
  - ThomPete 12y ago
    It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.
    - Retra 12y ago
      It is far harder to step on ants when you are designed to avoid stepping on them. Which naturally, those ants will try to do.
      - ThomPete 12y ago
        Which require the ants understand what kind of "steps" it can take which they won't.
        12y ago
- alexvr 12y ago
  My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.
steve-benjamins 12y ago
Philosophically I think it's a mistake to believe there is any possibility of creating artificial human intelligence (and the mistake is in our assumptions of what human intelligence is— computers are not human beings and human beings are not computers) ... But I've got to hand it to Elon Musk— when he believes in something he commits to doing something about it. EDIT - Saying I disagree that AI is a foregone conclusion on Hacker News is probably a bit trollish... Unfortunately I'm not smart enough to condense an explanation to this comment box— but my disagreement stems from philosophers such as Hubert Dreyfuss (What Computers Can't Do) who are working off of Heidegger and phenomenology. EDIT 2 - I know how douchey it is to namedrop Heidegger, but I really believe in this case it's necessary ...
- jsutton 12y ago
  Why wouldn't there be any possibility of artificial human intelligence? Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. There's nothing inherently preventing us from recreating that.
  - psycr 12y ago
    > recreating And that's the crux of the question isn't it? How do we determine if the intelligence is recreated or merely mimicked? Does such a distinction matter? Searle says yes. More here if you're interested: http://plato.stanford.edu/entries/chinese-room/ http://plato.stanford.edu/entries/chinese-room/
    - ajuc 12y ago
      This is misleading argument - there IS a Chinese speaker in the room, but it's not the person that executes the program - it's the system "person+program" that understands Chinese.
      - henrikschroder 12y ago
        The whole point of the argument is that the system doesn't understand Chinese. The system doesn't know what each sentence means, they don't evoke feelings, memories, images in the person in the room, there is no analysis, no agreement or disagreement with what is being said, it's just mechanical lookup.
creack 12y ago
I love the date of the article.
- utopiah 12y ago
  Scary isn't it. They have an AI AND a time machine...
deleted 12y ago
[deleted]
bhouston 12y ago
Is FutureOfLife.org a competitor to Machine Intelligence Research Institute? So now we have two charitable organizations researching how to make AI's beneficial? Who knew this was going to be a growth industry.
- nbouscal 12y ago
  They're not competitors so much as allied organizations using different methods, as far as I can tell.
- deleted 12y ago
  [deleted]
- eli_gottlieb 12y ago
  They're not so much competitors as FoL and FoI institutes are exploiting the comparative advantage of having vastly more famous and reputable staff who can be taken seriously by the academic community at large. I hang out on LessWrong sometimes and know some of the MIRI people, and while I appreciate the technical work they do on their subject, what they simply don't have is brand image. So they've gladly let someone with more brand-image than them take the "marketing and advocacy" parts of their cause off their hands, freeing the math nerds to concentrate on math.
- Kutta 12y ago
  Tallinn, Tegmark and Bostrom are long-time MIRI supporters, so this probably won't be positioned as a competitor. It also seems that there was a not-very-advertised AI conference in January, with pretty much everyone in the AI-risk scene attending, most of MIRI staff and Elon Musk included.
olssy 12y ago
If only humans could agree on what makes things beneficial to themselves. Are self-driving cars and rockets to Mars beneficial to humanity? Depending on how you look at it the answer is both yes and no.
sgt 12y ago
Where is Ray Kurzweil's name on that list?
- DanAndersen 12y ago
  From what I understand, Kurzweil tends to be much more optimistic about AI, predicting that humanity will merge with advancing AI rather than it running away from us.
  - Chathamization 12y ago
    And predicting that it’d happen five years ago…
CodeCube 12y ago
I know it's a bit dramatic at this point to take this stance ... but this almost gives me deja-vu of racism. Like a modern version of: http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Rock_integration_protest.jpg http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Ro... Here's what I mean; I'm not suggesting that AI has, or will have, some innate set of "rights". Nor am I suggesting that anyone is wrong for wanting to fund and research "AI Safety". It's just the first thing that came to mind while reading that post, as it spoke about need to steer, control, and regulate AI so that it's beneficial for "us" ... already setting up an "us" vs "them" dynamic. Anyways, just thought it was an interesting juxtaposition to mention ... thoughts?
- ObviousScience 12y ago
  I think it's incredibly selfish and poorly thought out to create another kind of mind only if it serves us. Much like parenting, I think it should be done because there's another kind of mind to be had, regardless of the exact outcome (which we shouldn't necessarily try to fully control).
  - eli_gottlieb 12y ago
    Parents do not routinely expect to be murdered by their children, nor is moral to expect or require such.
    - ObviousScience 12y ago
      About 5 parents are killed in the US a week. Of course, this isn't a HUGE danger, I'm just saying that it's a risk to having children. (This isn't counting things like mothers dying from childbirth, which still also happens.) My point wasn't that we should expect it to happen, but rather, neurotically making sure a child never could (or attempting to) rather than just having a child, raising it, and hoping for the best is bad parenting. As a secondary point: it's only humans that don't expect to be regularly murdered by their children; in other species, such an act is a routine occurrence. There's no particular reason to think that we're a super-special species in the grand scheme of things, and it's entirely possible that doing something like siring an artificial species better than us would end in our deaths. That doesn't, by default, make it not worth doing.
buro9 12y ago
The default assumption seems to be that if we create a thing of superior intellect, then that thing is going to be a threat to humans. I believe that it is possible to create intelligent machines, and that those machines will challenge our perception of what it means 'to be'. I don't necessarily agree with the implicit assumption that when that time comes, that a thing of superior intellect is obviously going to be 'evil' towards things of lesser intellect. Unless we are actually making a statement extrapolated from our own experience of how we perceive our relation to other life-forms on this planet.
- drdeca 12y ago
  I don't think the common claim is that it would be "evil" or malicious, but that it could be a threat, being both powerful and unpredictable. The idea that I've often seen people mention, is the idea of a "paperclip optimizer", in which an AI superintelligence primarily optimizes for there being more paperclips, not taking into account side effects of this, which other people might find harmful. If I understand the idea correctly, which I very well might not, it is meant to suggest that if an AI superintelligence is created with a particular goal in mind, we would do well to be very careful when choosing the AI's goals. It would I suppose be essentially a similar problem to that of a "literal genie". (Except harsher, because the AI would go through steps to accomplish the task, instead of just magically making the task so, and therefore would have more side effects?)
listic 12y ago
A post from the future! http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zps6ddff834.png http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zp...
k__ 12y ago
He fears Roko's basilisk ;) http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk
- bhouston 12y ago
  A theory that those that do not donate to AI-promoting groups will be punished in the future by AIs? Whoa, weird. Great for promoting fundraising activities though. Very church like.
  - butwhy 12y ago
    And Musk will be punished for donating to a cause potentially preventing its inception. ;)
    - LLWM 12y ago
      Actually, the argument is that an identical copy of Musk will be created and that copy will be punished. And for some reason, Musk should care.
  - JamesArgo 12y ago
    No one has every believed in the basilisk, save for maybe Yud. Everyone on LessWrong thought it was dumb from day one. LessWrong, now mostly dead, was a fun community full of smart people. That article is trying to make the place out as a cult, but no one bought any of that stuff, not even Roko. He was just playing around with local memes. It's pretty gross how this whole basilisk thing has turned out.
    - davidgerard 12y ago
      No, in the comments on the original basilisk post, Roko said he was in fact going through with the quantum gambling scheme that the post outlines. http://basilisk.neocities.org/ http://basilisk.neocities.org/ ctrl-F to "23 July 2010 06:52:40PM"
Keyframe 12y ago
I thought he was strapped for cash. Divorce and all.
- zodiakzz 12y ago
  Feeding the troll but he had his wife sign a contract where she can't claim ownership of any of his assets.. smart ;)
  - Keyframe 12y ago
    Not a troll :) I remember these headlines: http://venturebeat.com/2010/05/27/elon-musk-personal-finances/ http://venturebeat.com/2010/05/27/elon-musk-personal-finance... I wasn't paying attention much since then, so was wondering.
- mikeyouse 12y ago
  Elon's stakes in his various companies: - Tesla - 28M shares @ $193 = $5.4B - Solar City - 20M shares @ $51 - $1.0B - SpaceX - Still private, but some wild assumptions that Elon owns ~33% at a valuation of 4x total funding - $350M So his net worth is somewhere near (and probably north of) $7 Billion. If his wife takes half, he'll still be able to donate plenty to AI research.
honeybooboo123 12y ago
I doubt he can really believe that dropping 10M on a project will somehow prevent Skynet from happening. AI will just get nasty, or it won't. No one knows.
- butwhy 12y ago
  Fine, it will reduce the likelihood of it "getting nasty".
  - honeybooboo123 12y ago
    How the hell would it reduce it either? If there was only one organization in the world doing AI, then you could maybe bribe them to take a break or something. The only way to guarantee AI safety would be to just stop developing it further, but that's not going to happen, so how the hell is his 10M supposed to keep AI safe?
bra-ket 12y ago
we already have The Three Laws of Robotics http://en.wikipedia.org/wiki/Three_Laws_of_Robotics http://en.wikipedia.org/wiki/Three_Laws_of_Robotics
- rfrey 12y ago
  You're probably joking, and my humour unit is impaired today. But in case you're not... Those laws are the requirements document - the $10mm is for the implementation.
  - deleted 12y ago
    [deleted]
- Symmetry 12y ago
  If you can define "harm" well enough for the first law to be useful then you've solved the problem and laws 2 and 3 are probably superfluous.
- deleted 12y ago
  [deleted]
- breuleux 12y ago
  A lot of Asimov's work deals with loopholes and inadequacies in these laws, though: for instance, what is a human being? Does surgery violate the first law? If a human is going to injure another human, can the first law be overridden? If a robot believes that some entity is not human, do they have the responsibility to check before injuring them? I mean, realistically, AI safety is not going to be achieved with a list of written laws. Heck, the way it's going, we wouldn't know how to enforce such a list at all: first we would need the AI to understand human languages, and then we would need the AI to care, that is to say, robots would need to know and follow the second law before you can tell them about it.
kbody 12y ago
How can you control the creation of code? This news sounds like "donating to keep software bug-free". If someone wants or even accidentally, they can create buggy code or "evil" AI.
- cjfont 12y ago
  Perhaps the incentive here is to educate coders as to what patterns to avoid which will have an inclination toward "evil" AI. This won't stop others for whom this is the goal.
- davmre 12y ago
  You can donate to keep software bug-free by funding research into safer programming languages (e.g., Rust), tools for formal verification and static analysis, testing frameworks, software engineering methodology, etc. Sure, someone could always ignore all the research advances and just write unsafe, untested C code. But for the most part, people want to eliminate bugs, so as the techniques for doing so mature, they will tend to become adopted in practice and certain types of bugs will become much less frequent. Similarly, there are technical research questions related to how to design an intelligent agent that is "controllable", in that it acts to achieve its given goals but not to such an extent that it would resist being switched off or reprogrammed with new goals. Making progress on answering these questions doesn't guarantee that any given developer will use whatever techniques are discovered. But insofar as pretty much everyone, even evil masterminds, wants to maintain control over their AIs, the availability of techniques for 'safe' AI will at least decrease the likelihood that a powerful AI is built without any safeguards.
reacweb 12y ago
AI is a buzzword that is coming more and more in car. Creating a mental association between AI and safety is a very smart PR idea.
pekk 12y ago
AI is just software. So any discussion about AI gone wrong is really a discussion about malware and other kinds of software which are used by people to do bad things. If my software runs a power plant for an evil facility (full of sharks with lasers), is it evil software? The evilness of the software is parasitic on the intent of its operators. But if the issue is software that goes badly wrong in unexpected ways, we are really just talking about software engineering again, segfaults and Ariane 5 and how to handle user passwords. Obviously if we plan to outsource big tasks to chaotic software that we have trustingly given power over, let's say, all the cruise missiles, that is less of a scary emergent singularity scenario and more a plain-old-stupid scenario.
ggreer 12y ago
Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. -- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies[1] A lot of people in this thread seem to be falling into the same attractor. They see that Musk is worried about a superintelligent AI destroying humanity. To them, this seems preposterous. So they come up with an objection. "Superhuman AI is impossible." "Any AI smarter than us will be more moral than us." "We can keep it in an air-gapped simulated environment." etc. They are so sure about these barriers that they think $10 million spent on AI safety is a waste. It turns out that some very smart people have put a lot of thought into these problems, and they are still quite worried about superintelligence as an existential risk. If you want to really dig into the arguments for and against AI disaster (and discussion of how to control a superintelligence), I strongly recommend Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. It puts the comments here to shame. 1. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/ http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
- ThomPete 12y ago
  I always think about it like this. If immaterial matter could turn into conscious beings like us how can anyone claim that we are the last step in that evolution.
deleted 12y ago
[deleted]
rl3 12y ago
> "Along with research grants, the program will also include meetings and outreach programs aimed at bringing together academic AI researchers, industry AI developers and other key constituents to continue exploring how to maximize the societal benefits of AI; ..." Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope. For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse). However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise. For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it). Humans instruct it to solve the problem of death and disease. It does so. Humans instruct it to solve the problem of crime. It does so. Humans instruct it to solve the problem of war. It does so. Humans instruct it to solve the problem of poverty. It does so. Humans instruct it to explore the universe and solve the most vexing existential questions. It does so. In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima. That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering? Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence. For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing. While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.
- eli_gottlieb 12y ago
deleted 12y ago
[deleted]
- soup10 12y ago
  Ai's can now easily beat the smartest chess players on earth which have dedicated their lives to mastering the game. They can drive cars. They can automatically translate foreign languages. They have vision processing capabilities that are starting to rival humans. They run assembly lines, they diagnose diseases, and they can analyze and query sets of data far larger than human memory to coherently answer natural language questions. We don't know what will be possible in the future, the ai tech of today is already pretty powerful and will only get more so. Top ai researchers who understand the field most clearly think caution is warranted. Thats reason enough..

127 points by cocoflunchy 12y ago

I wonder how the goal can be achieved? How would you prevent the development of AI which is so intelligent that it is dangerous for humanity? Perhaps there might simply be fundamental physical constraints which limit intelligence? Still seems that something much faster and bigger than a human brain would be possible, though. Maybe by containment? If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual.

blfr 12y ago
It's more a matter of values this AI will hold than its level of intelligence. Boström argues that containment isn't really possible.
- FatalLogic 12y ago
  Is there any sure way to stop the super intelligent AI from changing its own inherent values though? I guess a problem with containment is that unless it's mathematically proven to be impervious, the super intelligent AI could have figured out that it's being contained and started to learn about the real world and how to manipulate it, while the humans outside still think they're safe.
  - ThrustVectoring 12y ago
    Changing your inherent values is a poor way to go about achieving your current set of values. The biggest issue is ensuring that everything we care about gets encapsulated as a value of the AI. If not, we wind up with things like "set off a bomb in the basement to more efficiently get people out of a burning building."
  - JamesArgo 12y ago
    It's assumed a rational agent judges its actions by how likely said action are to align the world with their fundamental values. For this reason, it is argued, a rational agent will not change its terminal goals. For example, suppose we build an agent that desires only to make shoes. That is, its ideal universe would consist of all available matter being converted into sneakers. Would it realize this is stupid? No. It judges it actions and thoughts only by how many shoes they are likely to bring into being. Any thought against this would be judged, correctly, as a negative contribution to shoe production and so wouldn’t be implemented, and likely would remain unthought. The best paper on this by far it Basic AI Drives by Stephen M. OMOHUNDRO: http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf http://selfawaresystems.files.wordpress.com/2008/01/ai_drive... I urge you to read it. It really is a wonderful paper.
    - TeMPOraL 12y ago
      The real problem is that human values are incredibly complex and messy, and any deviation from them multiplied by the power of a superintelligence would most likely doom us forever.
  - eli_gottlieb 12y ago
    > Is there any sure way to stop the super intelligent AI from changing its own inherent values though? Yes: give it inherent values. >I guess a problem with containment is that unless it's mathematically proven to be impervious, AGI, when it eventually exists, must run according to the basic laws of learning. If you don't give it evidence that's more likely in possible-worlds with an external world than without one, it won't posit an external world. Stopping that information leak isn't that hard, provided you have full control over the agent-environment surface in the first place. It does, however, make your "agent" completely useless for anything beyond playing Atari games.
- rl3 12y ago
  > Boström argues that containment isn't really possible. He never argued that containment isn't possible, just that it's incredibly dangerous. In fact, in Superintelligence, he uses the term Oracle as a means of referring to a contained or "boxed" superintelligent AGI. Just because the risks of constructing an Oracle or Oracles are extremely high, it doesn't mean we should refrain from doing so. In fact, it's more safe than the alternatives: a Genie or Sovereign; superintelligences that are entirely uncontained, the latter of which operating in a hands-off fashion in accordance with its goal/value system. Given the current, seemingly intractable nature of the superintelligence control problem, one possible method in which we may accomplish safe, superintelligent AGI is to first operate many variations of potentially unsafe "Oracles" and interrogate them for information that may further our research into AGI safety, or perhaps even solve the issue entirely. Generally speaking you shouldn't trust an Oracle, but if you have many radically different designs operating in containment that are oblivious to each other, the commonality in their output when posed such questions may assist in our realization of safe AGI that can operate in an uncontained fashion.
pavel_lishin 12y ago
I'm sorry for bringing this up, because it's vaguely controversial in some circles, and worse, a huge time-sink on par with TVTropes, but Yudkowsky attempted to run an experiment to see whether someone could be convinced by an AI, using nothing but text, to release the AI into the world. Apparently, he succeeded - his subject elected to release the fake AI. The method wasn't made public, as far as I know, and it might all be a hoax, but it's something to think about - how do you trap and imprison something smarter than you?
- ceejayoz 12y ago
  > how do you trap and imprison something smarter than you? Jails seem to have this figured out just fine.
  - JamesArgo 12y ago
    The correct analogy would be a jail run by a bacterium.
    - Retric 12y ago
      Prison bars are dumber than a bacterium and they work just fine. Still, the idea that AI is suddenly X smarter than people is ridiculously naive. Intelligence does not fit on a linear scale. And being smarter does not change the correct answer.
      - TeMPOraL 12y ago
        The problem isn't with AIs dumber than people - it's with what happens when someone finally builds a smarter one. It doesn't matter how good are prison bars if you're smart enough to social-engineer your way out of the cell. And human criminals do escape from prisons every now and then.
      - jacquesm 12y ago
        This is more like a 'reverse hacker', instead of a brilliant hacker trying to get into a system it is a brilliant hacker trying to get out of a system, and in this case the hacker is likely vastly more brilliant than the defenders. The same rules apply: the hacker has to succeed only once, the 'jailers' have to succeed all the times. Predicted long term outcome: escape.
mentos 12y ago
What about limiting AI to a certain clock rate?
- kanzure 12y ago
  > What about limiting AI to a certain clock rate? Not everyone will make high-cognitive-ability software that follows your clock rate rules. (And I think this is a good thing.)
- FatalLogic 12y ago
  Hypothetically, the AI would cunningly manufacture a crisis in which you are forced to increase its clock rate to enable it to calculate a solution to some impending disaster.
- JulianMorrison 12y ago
  http://www.rifters.com/real/2009/01/iterating-towards-bethlehem.html http://www.rifters.com/real/2009/01/iterating-towards-bethle... It's a peculiarity of humans that we can't trade off time for IQ. Do not assume an AI would be similarly limited.
- rl3 12y ago
  I'm not sure why the parent was downvoted; limiting an AI's computational capacity is considered a stunting method, and although such methods are not in and of themselves solutions to the control problem, they may be tools that assist us in developing a solution. For example, we may desire to first study AGI by restricting it to levels of intelligence slightly below that of a human. To accomplish this, we may employ various stunting and/or tripwire methods to ensure the AGI develops within our parameters.
gradys 12y ago
If homomorphic encryption gets off the ground, could an AI hide itself in homomorphically encrypted cloud compute capacity, earning Bitcoin to fund its survival?
- eli_gottlieb 12y ago
  Yes, but that relies on Bitcoin holding water over the long term. This is not so sure.
  - TeMPOraL 12y ago
    It probably wouldn't need bitcoin; it could figure out ways to trick humans into paying for the servers or just hack its way into electronic transfers.
Synthpixel 12y ago
The institute and the open letter are about maximizing the societal benefits of AI by funding research that provides net positives to humanity. The goal isn't to prevent the development of potentially harmful technologies. I believe they intend to fund positive research so that researchers do not resort to projects that do not provide net good, for example, in fields like weaponry. Not that safely developing hyperintelligent AI isn't an interesting topic. But I don't think AI research is anywhere close to that stage, considering that programmers largely still have to hard-code any actions that an agent can perform.
- butwhy 12y ago
  " The goal isn't to prevent the development of potentially harmful technologies" - actually, the goal includes "avoiding potential pitfalls". One would assume that creating danger towards humanity would be a pitfall.
yoha 12y ago
About the idea of containing an AI → https://en.wikipedia.org/wiki/AI_box_experiment https://en.wikipedia.org/wiki/AI_box_experiment
evanspa 12y ago
"If it's possible to keep the super intelligent AI in a virtual environment, but it can't detect that it's virtual." They did this in a Star Trek TNG episode ("Ship in a Bottle"). Data accidentally endows a holodeck character with super-intelligence. The way the crew eventually defeated him was to trick him into thinking he had left the holodeck and was able to live normally in the outside world; but it was really just another virtual environment.
- pzxc 12y ago
  Actually, Geordi endows Professor Moriarty with super-intelligence, by mistakenly asking the holodeck/computer to create an adversary "capable of defeating Data" instead of "capable of defeating Holmes" as he intended. God, I'm such a nerd. :P
  - colordrops 12y ago
    So the holodeck computer must be insanely intelligent. Or, it turns out that the building blocks for intelligence are simple and deterministic.
  - deleted 12y ago
    [deleted]
  - evanspa 12y ago
    Yup. Good catch :)
- comrade1 12y ago
  He doesn't sound that smart...
Retra 12y ago
Intelligent doesn't imply dangerous. It doesn't imply _anything_ so long as it is underspecified. That is the core problem: what do we actually want, and how do we know it exists? We want something that thinks enough like a human to understand and appreciate our goals, but enough different from a human to not suffer our faults. And in asking for this, we already suppose the existence of an AI that fundamentally disagrees with how humans make decisions. This is the 'better morality' problem. If I say "I'm more moral than you", I have to do so by showing that _your_ morality is contradictory with _your _ actions; that you are a hypocrite. Otherwise, we will disagree on some course of action, and you will have no reason to believe your decisions are incorrect except by _assuming_ that I am more moral, which is what we are trying to establish. So we really want a computer that can point out our errors and suggest corrections. This is one (vague) foundation for artificial intelligence that doesn't easily permit 'evil geniuses' since, so long as this is what the computer is doing, it cannot be said to be dangerous. Overall, I find AI discussion to be incredibly primitive and naive. It's practically a religious debate except among a very very few. Most people are not very creative about addressing these kinds of issues, and it's depressing that people are tied up in so thoroughly in bad terminology and non-issues.
- ThomPete 12y ago
  It doesn't have to have intent to be dangerous thats the problem. Just like you don't mean to step on an ant but probably have often done that.
  - Retra 12y ago
    It is far harder to step on ants when you are designed to avoid stepping on them. Which naturally, those ants will try to do.
    - ThomPete 12y ago
      Which require the ants understand what kind of "steps" it can take which they won't.
      - 12y ago
alexvr 12y ago
My first instinct is to put it in a virtual world, but if it's smart enough, it could perhaps quickly realize the nature of its world and feign ignorance. Then, under the pretext of simply improving its own quality of life, it could tempt us by engineering awesome things that would help us tremendously in the real world -- all we would have to do is copy its work (because we wouldn't be able to fully comprehend it), and our energy problems become a thing of the past! But the AI forgot to mention that its wonderful creation also gives it access to our physical world and puts us at its mercy.

steve-benjamins 12y ago

Philosophically I think it's a mistake to believe there is any possibility of creating artificial human intelligence (and the mistake is in our assumptions of what human intelligence is— computers are not human beings and human beings are not computers) ... But I've got to hand it to Elon Musk— when he believes in something he commits to doing something about it. EDIT - Saying I disagree that AI is a foregone conclusion on Hacker News is probably a bit trollish... Unfortunately I'm not smart enough to condense an explanation to this comment box— but my disagreement stems from philosophers such as Hubert Dreyfuss (What Computers Can't Do) who are working off of Heidegger and phenomenology. EDIT 2 - I know how douchey it is to namedrop Heidegger, but I really believe in this case it's necessary ...

jsutton 12y ago
Why wouldn't there be any possibility of artificial human intelligence? Our intelligence is just a manifestation of a physical system of neurons that follow the laws of physics. There's nothing inherently preventing us from recreating that.
- psycr 12y ago
  > recreating And that's the crux of the question isn't it? How do we determine if the intelligence is recreated or merely mimicked? Does such a distinction matter? Searle says yes. More here if you're interested: http://plato.stanford.edu/entries/chinese-room/ http://plato.stanford.edu/entries/chinese-room/
  - ajuc 12y ago
    This is misleading argument - there IS a Chinese speaker in the room, but it's not the person that executes the program - it's the system "person+program" that understands Chinese.
    - henrikschroder 12y ago
      The whole point of the argument is that the system doesn't understand Chinese. The system doesn't know what each sentence means, they don't evoke feelings, memories, images in the person in the room, there is no analysis, no agreement or disagreement with what is being said, it's just mechanical lookup.

creack 12y ago

I love the date of the article.

utopiah 12y ago
Scary isn't it. They have an AI AND a time machine...

deleted 12y ago

[deleted]

bhouston 12y ago

Is FutureOfLife.org a competitor to Machine Intelligence Research Institute? So now we have two charitable organizations researching how to make AI's beneficial? Who knew this was going to be a growth industry.

nbouscal 12y ago
They're not competitors so much as allied organizations using different methods, as far as I can tell.
deleted 12y ago
[deleted]
eli_gottlieb 12y ago
They're not so much competitors as FoL and FoI institutes are exploiting the comparative advantage of having vastly more famous and reputable staff who can be taken seriously by the academic community at large. I hang out on LessWrong sometimes and know some of the MIRI people, and while I appreciate the technical work they do on their subject, what they simply don't have is brand image. So they've gladly let someone with more brand-image than them take the "marketing and advocacy" parts of their cause off their hands, freeing the math nerds to concentrate on math.
Kutta 12y ago
Tallinn, Tegmark and Bostrom are long-time MIRI supporters, so this probably won't be positioned as a competitor. It also seems that there was a not-very-advertised AI conference in January, with pretty much everyone in the AI-risk scene attending, most of MIRI staff and Elon Musk included.

olssy 12y ago

If only humans could agree on what makes things beneficial to themselves. Are self-driving cars and rockets to Mars beneficial to humanity? Depending on how you look at it the answer is both yes and no.

sgt 12y ago

Where is Ray Kurzweil's name on that list?

DanAndersen 12y ago
From what I understand, Kurzweil tends to be much more optimistic about AI, predicting that humanity will merge with advancing AI rather than it running away from us.
- Chathamization 12y ago
  And predicting that it’d happen five years ago…

CodeCube 12y ago

I know it's a bit dramatic at this point to take this stance ... but this almost gives me deja-vu of racism. Like a modern version of: http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Rock_integration_protest.jpg http://upload.wikimedia.org/wikipedia/commons/f/fa/Little_Ro... Here's what I mean; I'm not suggesting that AI has, or will have, some innate set of "rights". Nor am I suggesting that anyone is wrong for wanting to fund and research "AI Safety". It's just the first thing that came to mind while reading that post, as it spoke about need to steer, control, and regulate AI so that it's beneficial for "us" ... already setting up an "us" vs "them" dynamic. Anyways, just thought it was an interesting juxtaposition to mention ... thoughts?

ObviousScience 12y ago
I think it's incredibly selfish and poorly thought out to create another kind of mind only if it serves us. Much like parenting, I think it should be done because there's another kind of mind to be had, regardless of the exact outcome (which we shouldn't necessarily try to fully control).
- eli_gottlieb 12y ago
  Parents do not routinely expect to be murdered by their children, nor is moral to expect or require such.
  - ObviousScience 12y ago
    About 5 parents are killed in the US a week. Of course, this isn't a HUGE danger, I'm just saying that it's a risk to having children. (This isn't counting things like mothers dying from childbirth, which still also happens.) My point wasn't that we should expect it to happen, but rather, neurotically making sure a child never could (or attempting to) rather than just having a child, raising it, and hoping for the best is bad parenting. As a secondary point: it's only humans that don't expect to be regularly murdered by their children; in other species, such an act is a routine occurrence. There's no particular reason to think that we're a super-special species in the grand scheme of things, and it's entirely possible that doing something like siring an artificial species better than us would end in our deaths. That doesn't, by default, make it not worth doing.

buro9 12y ago

The default assumption seems to be that if we create a thing of superior intellect, then that thing is going to be a threat to humans. I believe that it is possible to create intelligent machines, and that those machines will challenge our perception of what it means 'to be'. I don't necessarily agree with the implicit assumption that when that time comes, that a thing of superior intellect is obviously going to be 'evil' towards things of lesser intellect. Unless we are actually making a statement extrapolated from our own experience of how we perceive our relation to other life-forms on this planet.

drdeca 12y ago
I don't think the common claim is that it would be "evil" or malicious, but that it could be a threat, being both powerful and unpredictable. The idea that I've often seen people mention, is the idea of a "paperclip optimizer", in which an AI superintelligence primarily optimizes for there being more paperclips, not taking into account side effects of this, which other people might find harmful. If I understand the idea correctly, which I very well might not, it is meant to suggest that if an AI superintelligence is created with a particular goal in mind, we would do well to be very careful when choosing the AI's goals. It would I suppose be essentially a similar problem to that of a "literal genie". (Except harsher, because the AI would go through steps to accomplish the task, instead of just magically making the task so, and therefore would have more side effects?)

listic 12y ago

A post from the future! http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zps6ddff834.png http://i298.photobucket.com/albums/mm249/hrenistic/feb_15_zp...

k__ 12y ago

He fears Roko's basilisk ;) http://rationalwiki.org/wiki/Roko%27s_basilisk http://rationalwiki.org/wiki/Roko%27s_basilisk

bhouston 12y ago
A theory that those that do not donate to AI-promoting groups will be punished in the future by AIs? Whoa, weird. Great for promoting fundraising activities though. Very church like.
- butwhy 12y ago
  And Musk will be punished for donating to a cause potentially preventing its inception. ;)
  - LLWM 12y ago
    Actually, the argument is that an identical copy of Musk will be created and that copy will be punished. And for some reason, Musk should care.
- JamesArgo 12y ago
  No one has every believed in the basilisk, save for maybe Yud. Everyone on LessWrong thought it was dumb from day one. LessWrong, now mostly dead, was a fun community full of smart people. That article is trying to make the place out as a cult, but no one bought any of that stuff, not even Roko. He was just playing around with local memes. It's pretty gross how this whole basilisk thing has turned out.
  - davidgerard 12y ago
    No, in the comments on the original basilisk post, Roko said he was in fact going through with the quantum gambling scheme that the post outlines. http://basilisk.neocities.org/ http://basilisk.neocities.org/ ctrl-F to "23 July 2010 06:52:40PM"

Keyframe 12y ago

I thought he was strapped for cash. Divorce and all.

zodiakzz 12y ago
Feeding the troll but he had his wife sign a contract where she can't claim ownership of any of his assets.. smart ;)
- Keyframe 12y ago
  Not a troll :) I remember these headlines: http://venturebeat.com/2010/05/27/elon-musk-personal-finances/ http://venturebeat.com/2010/05/27/elon-musk-personal-finance... I wasn't paying attention much since then, so was wondering.
mikeyouse 12y ago
Elon's stakes in his various companies: - Tesla - 28M shares @ $193 = $5.4B - Solar City - 20M shares @ $51 - $1.0B - SpaceX - Still private, but some wild assumptions that Elon owns ~33% at a valuation of 4x total funding - $350M So his net worth is somewhere near (and probably north of) $7 Billion. If his wife takes half, he'll still be able to donate plenty to AI research.

honeybooboo123 12y ago

I doubt he can really believe that dropping 10M on a project will somehow prevent Skynet from happening. AI will just get nasty, or it won't. No one knows.

butwhy 12y ago
Fine, it will reduce the likelihood of it "getting nasty".
- honeybooboo123 12y ago
  How the hell would it reduce it either? If there was only one organization in the world doing AI, then you could maybe bribe them to take a break or something. The only way to guarantee AI safety would be to just stop developing it further, but that's not going to happen, so how the hell is his 10M supposed to keep AI safe?

bra-ket 12y ago

we already have The Three Laws of Robotics http://en.wikipedia.org/wiki/Three_Laws_of_Robotics http://en.wikipedia.org/wiki/Three_Laws_of_Robotics

rfrey 12y ago
You're probably joking, and my humour unit is impaired today. But in case you're not... Those laws are the requirements document - the $10mm is for the implementation.
- deleted 12y ago
  [deleted]
Symmetry 12y ago
If you can define "harm" well enough for the first law to be useful then you've solved the problem and laws 2 and 3 are probably superfluous.
deleted 12y ago
[deleted]
breuleux 12y ago
A lot of Asimov's work deals with loopholes and inadequacies in these laws, though: for instance, what is a human being? Does surgery violate the first law? If a human is going to injure another human, can the first law be overridden? If a robot believes that some entity is not human, do they have the responsibility to check before injuring them? I mean, realistically, AI safety is not going to be achieved with a list of written laws. Heck, the way it's going, we wouldn't know how to enforce such a list at all: first we would need the AI to understand human languages, and then we would need the AI to care, that is to say, robots would need to know and follow the second law before you can tell them about it.

kbody 12y ago

How can you control the creation of code? This news sounds like "donating to keep software bug-free". If someone wants or even accidentally, they can create buggy code or "evil" AI.

cjfont 12y ago
Perhaps the incentive here is to educate coders as to what patterns to avoid which will have an inclination toward "evil" AI. This won't stop others for whom this is the goal.
davmre 12y ago
You can donate to keep software bug-free by funding research into safer programming languages (e.g., Rust), tools for formal verification and static analysis, testing frameworks, software engineering methodology, etc. Sure, someone could always ignore all the research advances and just write unsafe, untested C code. But for the most part, people want to eliminate bugs, so as the techniques for doing so mature, they will tend to become adopted in practice and certain types of bugs will become much less frequent. Similarly, there are technical research questions related to how to design an intelligent agent that is "controllable", in that it acts to achieve its given goals but not to such an extent that it would resist being switched off or reprogrammed with new goals. Making progress on answering these questions doesn't guarantee that any given developer will use whatever techniques are discovered. But insofar as pretty much everyone, even evil masterminds, wants to maintain control over their AIs, the availability of techniques for 'safe' AI will at least decrease the likelihood that a powerful AI is built without any safeguards.

reacweb 12y ago

AI is a buzzword that is coming more and more in car. Creating a mental association between AI and safety is a very smart PR idea.

pekk 12y ago

AI is just software. So any discussion about AI gone wrong is really a discussion about malware and other kinds of software which are used by people to do bad things. If my software runs a power plant for an evil facility (full of sharks with lasers), is it evil software? The evilness of the software is parasitic on the intent of its operators. But if the issue is software that goes badly wrong in unexpected ways, we are really just talking about software engineering again, segfaults and Ariane 5 and how to handle user passwords. Obviously if we plan to outsource big tasks to chaotic software that we have trustingly given power over, let's say, all the cruise missiles, that is less of a scary emergent singularity scenario and more a plain-old-stupid scenario.

ggreer 12y ago

Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. -- Nick Bostrom, Superintelligence: Paths, Dangers, Strategies[1] A lot of people in this thread seem to be falling into the same attractor. They see that Musk is worried about a superintelligent AI destroying humanity. To them, this seems preposterous. So they come up with an objection. "Superhuman AI is impossible." "Any AI smarter than us will be more moral than us." "We can keep it in an air-gapped simulated environment." etc. They are so sure about these barriers that they think $10 million spent on AI safety is a waste. It turns out that some very smart people have put a lot of thought into these problems, and they are still quite worried about superintelligence as an existential risk. If you want to really dig into the arguments for and against AI disaster (and discussion of how to control a superintelligence), I strongly recommend Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. It puts the comments here to shame. 1. http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/ http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

ThomPete 12y ago
I always think about it like this. If immaterial matter could turn into conscious beings like us how can anyone claim that we are the last step in that evolution.

deleted 12y ago

[deleted]

rl3 12y ago

> "Along with research grants, the program will also include meetings and outreach programs aimed at bringing together academic AI researchers, industry AI developers and other key constituents to continue exploring how to maximize the societal benefits of AI; ..." Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope. For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse). However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise. For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it). Humans instruct it to solve the problem of death and disease. It does so. Humans instruct it to solve the problem of crime. It does so. Humans instruct it to solve the problem of war. It does so. Humans instruct it to solve the problem of poverty. It does so. Humans instruct it to explore the universe and solve the most vexing existential questions. It does so. In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima. That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering? Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence. For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing. While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.

eli_gottlieb 12y ago

deleted 12y ago

[deleted]

soup10 12y ago
Ai's can now easily beat the smartest chess players on earth which have dedicated their lives to mastering the game. They can drive cars. They can automatically translate foreign languages. They have vision processing capabilities that are starting to rival humans. They run assembly lines, they diagnose diseases, and they can analyze and query sets of data far larger than human memory to coherently answer natural language questions. We don't know what will be possible in the future, the ai tech of today is already pretty powerful and will only get more so. Top ai researchers who understand the field most clearly think caution is warranted. Thats reason enough..