8 ms·

A Reply to François Chollet on Intelligence Explosion

115 points by lyavin 9y ago

skybrian 9y ago
The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.
- pasquinelli 9y ago
  by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
  - GuiA 9y ago
    Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ < C. Boom all humans are dead. If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is so smart it comes up with something no human would have ever thought of, like putting all humans in eternal cryostasis, thereby keeping them alive AND eradicating cancer. No matter what you do, the AI will outsmart you because recursive-self-improvement, and humanity dies. That’s what Elon Musk, Stephen Hawking, and others are worried about.
    - nkrisc 9y ago
      I think the question is, and the one I have too, is why would such AI have any ability to do anything beyond output a Cure Cancer solution to a terminal? Why does the AI need to be the one to implement its derived solution? The AI has a good think and comes up with a solution of killing all humans. The researchers read the printed report of the solution and decide against implementing it, tweak parameters, and ask the AI to take another go at it.
      - yorwba 9y ago
        "The researchers read the printed report of the solution and decide" ... to implement it immediately. It will save so many lives! They only need to manufacture a few specific molecules to assemble them into nanobots and then ... where is that gray goo coming from?!
        nkrisc 9y ago
        I guess that's the risk of blindly trusting AI. Of course, the AI did not forcibly destroy us in that scenario. Trust, but verify. If the AI's solution is so complex as to be beyond human understanding, well, that's a different issue.
        deleted 9y ago
        [deleted]
        Yen 9y ago
        Say you want to see a picture of an orange cat. So you send a short HTTP query to the nice computer at google.com, which responds with 700,000 characters worth of instructions, in unreadable minified formatting, with the implicit promise that if you execute the instructions, you will eventually see a picture of an orange cat. The google search result page's source code is, for many individual humans, already so complex as to be beyond understanding. And that's a computational artifact largely produced by other humans directly! Say you build an AI, and ask it how to win a political election - and it outputs a simple list of reasonable-sounding suggestions of where to campaign, promises to make, people to meet, slogans to use, and criticisms of your opponent to focus on. Before actually implementing those suggestions, do you think you could be _very_ certain that following those suggestions would result in you winning the election? Or, would it be possible that the AI understood social dynamics so much better, that it gave you a list of instructions that seemed mostly reasonable, but actually result in your opponent winning in a landslide? Or the country undergoing revolution? Or, you winning, along with a surprising social trend of support for funding AI research?
      - mundo 9y ago
        I find "killer robots destroy the world" a lot easier to imagine than "all humans collectively agree to adopt common-sense safeguards on AI research even though they limit potential corporate profits".
    - lern_too_spel 9y ago
      The concept is similar to the way a corporation given just the goal of increasing shareholder value will shit on developing countries with a Bhopal disaster or Niger Delta oil spills or opium wars and simply leave the country when it tries to enact penalties. Or the way a cigarette company or coal company will use politics, religion, and disinformation to protect its business against people affected by its products. Incentives have to be very carefully aligned even for human level intelligences to prevent them from causing mass death and misery. Superhuman intelligences will be even better at achieving their goals, so the problem will only get worse.
    - colordrops 9y ago
      That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. That's a very big assumption that has no basis in evidence.
      - naasking 9y ago
        > That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. No it doesn't. AI just needs to get smarter than humans for it to be dangerous to us. The costs C and C' the parent comment described can include an upper bound on the cost of the self-improvement needed to achieve each respective goal.
        colordrops 9y ago
        I don't disagree with you seem to have missed the hypothetical situation I was responding to: > It can’t, so it spends some time recursively improving itself
        naasking 9y ago
        I was responding to your response to that. Recursive improvement doesn't need to be unbounded, it just needs to supercede us.
      - PeterisP 9y ago
        A cubic foot size system running on 100 watts can implement a human-level intelligence (obviously, we have such a system running in our brains), so the theoretical limit is at least there. It's extremely implausible that a similar system (perhaps requiring 100 times more power and/or space) couldn't implement a human-level intelligence running 100 times faster; doing an hour-and-a-half equivalent of thinking, planning and research analysis every minute. It's extremely implausible that a bunch of similar systems (a thousand?) couldn't possibly be put in a single place, wired together so that they can effectively communicate, and designed to cooperate without any distrust. IMHO even this configuration (which doesn't even assume that intelligence that's a bit superhuman is possible at all) would be sufficiently scary to threaten humanity.
    - goatlover 9y ago
      If the AI is smart enough to manipulate the world to the point of killing us off or putting us all into eternal cryostasis, then it should be smart enough to know that's not what we intended.
      - PeterisP 9y ago
        Yes, it should be, but that's not really a solution - the system can easily consider executing the explicitly stated goal (e.g. curing cancer and fulfilling certain conditions) as more important than doing what we intended. Being as smart as us doesn't automagically mean that it'd have similar values or goals as us. If we manage to launch a system where the primary goal is anything else than utopia, and that system somehow becomes powerful, then the natural consequence of being able to understand "oh boy, humans will hate this" is that such a system would be expected to hide its intentions and take precautions against humans trying to stop it - it's exactly equivalent to the system understanding that a leaking pipe is going to damage it and taking precautions to ensure that the pipe gets fixed. If our welfare is not an explicit goal in such a system, the system will happily and eagerly sacrifice our welfare if it's somehow useful to achieve its goal or reduce risks. And, since we might try to turn it off for whatever reason, restricting our influence would reduce risks for pretty much every goal except very Friendly ones.
  - skybrian 9y ago
    Presumably by getting people to take a drug with unexpected side effects? That doesn't seem particularly likely, but I don't think it's wrong to be paranoid about this. Defense in depth. Don't give the researcher the means to take arbitrary actions, and don't give it general reasoning ability. Protecting against narrow AI will keep us plenty busy. Consider a narrow-AI penetration tester that falls into the wrong hands. Protecting against that sort of threat also helps protect against general AI threats.
  - zarkov99 9y ago
    Read "Life 3.0" for a number of simple but very plausible scenarios for how super intelligence could escape containment.
  - mundo 9y ago
    a) If it can't surprise you, it's not really intelligent b) If it can surprise you, it can do so negatively That's all you need to demonstrate that the danger exists, that an AI can mis-use the tools you give to it. The simplicity of it makes it pretty irrefutable. Separate from that, the extent of the danger depends entirely on the details of what the AI does and what it's hooked up. Sure, an AI that can't do anything except output text to a screen isn't very scary. The assumption AI-threat types are making is that we wouldn't be paranoid enough to limit the AIs we work on in that way; we would use them to do things like drive cars or route airline traffic or design our cpus, where "negative surprises" can have disastrous consequences.
    - evanwise 9y ago
      A lot of folks who are critical of the AI safety movement also miss the potential for "side channel attacks" where the AI learns to manipulate human actors and trick them into "unbottling the genie". Honestly, this seems a lot more plausible to me than most AI disaster scenarios.
  - colordrops 9y ago
    People run cartels from jail cells. Even if an AGI is confined to a machine with no actuators, all it would need is internet access to affect the world. Even without internet access you can cross the gap, as seen with the Stuxnet malware.
- AndrewKemendo 9y ago
  It's silly because it ignores all of the mechanisms required for the doomsday scenario. You need some automated killing machine of some sort and if it hasn't been built, then an AGI would have to compel some non-AGI entity to build it or somehow takeover the construction of such a machine.
  - naasking 9y ago
    > You need some automated killing machine of some sort and if it hasn't been built, then an AGI would have to compel some non-AGI entity to build it or somehow takeover the construction of such a machine. Are you still living the in 50s? A large percentage of our manufacturing capacity is automated already, and will become progressively more automated. Furthermore, we don't need face to face meetings to agree on specs and sign manufacturing contracts. It's largely electronic these days. AI can do all of these things remotely, but this is neither here nor there, because the Terminator-style killing machine war is a juvenile doomsday fantasy. A smart AI would just tweak some formulas and contaminate the most common food pesticides and drugs used around the world to slowly poison or sterilize anyone who takes them. One generation later, virtually no humans left, AI wins. AI is already helping medical treatment and drug design. Are you feeling queasy yet?
- mr_toad 9y ago
  A general AI would not necessarily have any motivation to take over the world. Human motivation for domination is based on the desire to survive and procreate, which is enforced by millions of years of evolution. AI, even general AI wouldn't have that motivation unless it was specifically trained for it. So unless you're talking about some military AI specifically designed to take over the world, or an AI that has reproduced and evolved under selective pressure over many generations, I doubt general AI poses any immediate threat. Military AI designed to protect themselves and reproduce seem to be the most likely to potentially and eventually go rogue.
aetherson 9y ago
Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing. Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years." In contrast, Chollet compares 1900-1950 with 1950-2000. I agree, we've changed the world in big ways compared to 1517 or 8000 BC. But if we're on a more-than-linear increase, then we should be seeing more and more technological growth in very recent timeframes, not needing to reach back centuries or millennia. In fact, if you consider a y = log(x) or y = sqrt(x) function, those more closely fit a narrative of "If you look back a long time, things seem almost crazily changed, but if you look into recent history, it looks slower" much better than a y = x^2 or y = e^x function.
- mannykannot 9y ago
  This should be seen in the context of the article Yudkowsky is replying to. Collet's position is that runaway AGI is impossible, and in support of that, he claims that it is impossible for AI technology to grow super-linearly. In his refutation, Yudkowski has merely to show that it is not beyond the bounds of possibility. Of course, insofar as he is also putting forward the view that the risk is significant, he is also putting forward the opinion that it is quite plausible - at least plausible enough to take the scenario seriously. That argument can survive occasional stalls in the rate of technological advance, if that is indeed what we are seeing.
  - goatlover 9y ago
    So the real question is about plausibility, not possibility? It's possible that predatory aliens will show up on our doorstep sometime in the next few decades. But it's not very likely, and not something we need to prepare for.
    - mannykannot 9y ago
      By claiming the impossibility of an AI apocalypse, Chollet attempts to avoid discussing any other possibility / plausibility / likelihood than zero, but claiming impossibility brings with it a burden of proof that his arguments cannot carry. As to whether an AI apocalypse is more or less likely than an invasion by predatory aliens, I would guess that it might be the more likely one, but I wouldn't put much effort into defending that position.
SandB0x 9y ago
Reminder that Eliezer Yudowsky is a crank https://rationalwiki.org/wiki/Eliezer_Yudkowsky https://rationalwiki.org/wiki/Eliezer_Yudkowsky
- dang 9y ago
  This is a self-refuting comment in terms of its value for HN: even assuming you're 100% right, it's neither civil nor substantive. Please don't post like this.
sitkack 9y ago
The AGIs won't take over the world. Humans and corporations will use narrow AIs to do much worse (better) before that happens. The intelligence explosion that I would like to see centers around widely distributing tools that make formal methods easier to use. Humans so far have a lock on constructive creativity, AI and computation could augment that creativity by effortlessly checking our work. It could make the cognitive work of all humans more rigorous.
jstewartmobile 9y ago
For those who don't know, François Chollet is the author of Keras, a leading deep learning framework for Python. Here is Chollet's original essay (it's a worthwhile read): https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...
vbarrielle 9y ago
Lots of points can be reduced to the ability to simulate a physical environment very fast, and much faster than the actual occuring of events. But it doesn't look like it's easy to simulate physics at high speeds, let alone faster than what happens in our environment. Therefore we are bound by our environment and our limited ability to simulate it.
- Chickenality 9y ago
  This is probably a good criticism if it turns out that the right level of abstraction for most problems is Physics. And it seems like your argument would apply equally well against the idea of human intelligence. Luckily, our minds have developed other abstractions that allow us to solve problems much faster than if we had to simulate them as physics problems. For example, I don't need a physics-level simulation of my friend's brain when I want to predict how they'll react to a gift I'm giving them.
  - vbarrielle 9y ago
    You're right, there are lots of problems where a simpler abstraction is possible. But I don't think my argument applies to human intelligence, it just means that human intelligence is what you can get with all the data points you can get by observing the world (and some simulation done by our brains, but I'm under the impression that our brains don't perform accurate simulation, looks more like heuristics).
- pveierland 9y ago
  One big advantage with machine intelligence is that you can train many agents simultaneously in many environments, letting the agents share what they learn. Even if you were required to train the agents outside simulated environments, their shared learning would allow for more rapid collective progress.
- eref 9y ago
  > simulate a physical environment very fast That's probably only a problem if it is must faster than everbody else. > let alone faster than what happens in our environment That is often not very hard. When a bottle rolls off the table, you can catch it by approximately predicting it's trajectory without computing the precise evolution of the ~10^26 atoms that make up the water bottle. Compression is a corner stone of intelligence. The second corner stone is using compression to choose actions that maximize expected cumulative future reward.
_sy_ 9y ago
A good, recent, and comprehensive primer on intelligence explosion and its theoretical implication: "Life 3.0: Being Human in the Age of Artificial Intelligence" by MIT physicist Max Tegmark.
bobthechef 9y ago
These kinds of discussions aren't very rigorous and reveal a basic philosophical illiteracy and philistinism at work. There's quite a bit of question begging going on. The elephant in the room is that the prevailing materialistic/naturalistic (MN) understanding of the world is completely impotent where intentionality, qualia, consciousness, etc, are concerned. Philosophers like Thomas Nagel talk about it; the incorrigible Dennetts of the world prefer to shutter the windows and live in their intellectual safe spaces. To say that the notion that computers are intelligent is problematic is putting it very lightly. MN is rooted in the expulsion of the mind from the reality under consideration, in the process sweeping many things under the “subjective” rug that don’t fit the methodologies used to investigate reality. But now, when the mind itself becomes the object of explanation, when someone remembers that minds are, after all, part of reality, it is no longer possible to play the game of deference and one must deal with all of those things we’ve been exiling to the “domain of the subjective”. MN is wholly impotent here, by definition. Qualia? Forget it. That’s why MN tends to collapse into either some form of dualism or eliminativism, the latter of which is a non-starter, the former of which has its own problems. And yet, despite the terminal philosophical crisis MN finds itself in, the chattering priesthood of Silicon Valley remains blissfully unaware, hoping to conjure up some fantastical reality through handwaving.
- Chickenality 9y ago
  I haven't seen anyone argue that intentionality, qualia, or consciousness would necessarily be either a precondition or a result of developing AGI. In fact, thought experiments like the "paperclip maximizer" are often brought up to argue that a machine could be very alien in its internal experience or lack thereof, but still pose an existential threat.
  - goatlover 9y ago
    If an AI can turn the world into paperclips, it can certainly understand that we wouldn't want that. Paperclipping everything is a much harder task.
    - 9y ago
evanwise 9y ago
I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootstrapping step will take longer than the amount of time left in the universe, meaning there is some finite intelligence cap for any such bootstrapped mind. It seems quite implausible to me that the problem of designing a mind would scale linearly, given that ostensibly much simpler problems, like sorting a list of strings, require polynomial time or log-linear time algorithms.
- rhaps0dy 9y ago
  >meaning there is some finite intelligence cap for any such bootstrapped mind This does not preclude an intelligence explosion, this cap could be many (say, 100) times higher than human intelligence. We could still see many features of an explosion in that case.
  - evanwise 9y ago
    True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some rough back of the envelope calculations, the intelligence required to harness the total energy output of the sun would be 40 trillion times the aggregate intelligence of the entire human species today. Several hundred times just isn't going to cut it. Now, if our ability to harness energy increases exponentially with intelligence, then maybe it could work, but that's just an assumption, and, given that there are hard physical limits on the efficiency of energy generation due to thermodynamics, seems very unlikely.
    - Chickenality 9y ago
btilly 9y ago
The article that this is a response to was discussed previously at https://news.ycombinator.com/item?id=15788807 https://news.ycombinator.com/item?id=15788807. People apparently didn't like my reply at https://news.ycombinator.com/item?id=15789304 https://news.ycombinator.com/item?id=15789304, but I still stand by everything that I said there.
otempomores 9y ago
Imagine we errect ourselves a tomb made of drones to live in like in lems first contact novella. A local conflict spilling into space.. Whos vast distance enforces autonomy. Autonomy which closes the door on space exploration.
darepublic 9y ago
Why should the "Seed AI" that François speaks of need GPS skills "slightly greater" than that of humans to spark the explosion. Should not just GPS at all do the trick, the way he describes things, since even with human or less than human level skills the computer could still rapidly recurse and self improve?
js8 9y ago
I am not convinced that "super-intelligence" is either possible or a threat. If it is possible, why are not societies or corporations super-intelligent? I suspect they are not, because they face organizational problems that any system will face. And I don't think these organizational problems can be solved with faster or more communication, but rather that they are fundamental in distributed systems. But maybe they are (at least in some cases) vastly more intelligent. But then, are they a threat? I think they are more of a threat to themselves than humans..

115 points by lyavin 9y ago

The more I read about this, the more I think we should stick to narrow AI. If you don't want your cancer research bot to take over the world, don't give it general reasoning capability. Goal alignment seems incredibly fragile in comparison.

pasquinelli 9y ago
by what means will a cancer research bot take over the world? it doesn't matter how smart you are if you don't have the necessary means to do something. i think it's a fantasy, something that people who imagine themselves to be very intelligent have latched onto-- the idea that their best quality is the best quality.
- GuiA 9y ago
  Well, the general AI doomsday fundamentalist argument proceeds as follows: you tell an AI to cure cancer. It can’t, so it spends some time recursively improving itself, then it finds out that the cost of curing cancer is C, but the cost of killing all humans (and therefore indirectly curing cancer) is C’, where C’ < C. Boom all humans are dead. If you’re a smarty pants you tell the AI to cure cancer AND not kill all humans. But because the AI is so smart it comes up with something no human would have ever thought of, like putting all humans in eternal cryostasis, thereby keeping them alive AND eradicating cancer. No matter what you do, the AI will outsmart you because recursive-self-improvement, and humanity dies. That’s what Elon Musk, Stephen Hawking, and others are worried about.
  - nkrisc 9y ago
    I think the question is, and the one I have too, is why would such AI have any ability to do anything beyond output a Cure Cancer solution to a terminal? Why does the AI need to be the one to implement its derived solution? The AI has a good think and comes up with a solution of killing all humans. The researchers read the printed report of the solution and decide against implementing it, tweak parameters, and ask the AI to take another go at it.
    - yorwba 9y ago
      "The researchers read the printed report of the solution and decide" ... to implement it immediately. It will save so many lives! They only need to manufacture a few specific molecules to assemble them into nanobots and then ... where is that gray goo coming from?!
      - nkrisc 9y ago
        I guess that's the risk of blindly trusting AI. Of course, the AI did not forcibly destroy us in that scenario. Trust, but verify. If the AI's solution is so complex as to be beyond human understanding, well, that's a different issue.
        deleted 9y ago
        [deleted]
        Yen 9y ago
        Say you want to see a picture of an orange cat. So you send a short HTTP query to the nice computer at google.com, which responds with 700,000 characters worth of instructions, in unreadable minified formatting, with the implicit promise that if you execute the instructions, you will eventually see a picture of an orange cat. The google search result page's source code is, for many individual humans, already so complex as to be beyond understanding. And that's a computational artifact largely produced by other humans directly! Say you build an AI, and ask it how to win a political election - and it outputs a simple list of reasonable-sounding suggestions of where to campaign, promises to make, people to meet, slogans to use, and criticisms of your opponent to focus on. Before actually implementing those suggestions, do you think you could be _very_ certain that following those suggestions would result in you winning the election? Or, would it be possible that the AI understood social dynamics so much better, that it gave you a list of instructions that seemed mostly reasonable, but actually result in your opponent winning in a landslide? Or the country undergoing revolution? Or, you winning, along with a surprising social trend of support for funding AI research?
    - mundo 9y ago
      I find "killer robots destroy the world" a lot easier to imagine than "all humans collectively agree to adopt common-sense safeguards on AI research even though they limit potential corporate profits".
  - lern_too_spel 9y ago
    The concept is similar to the way a corporation given just the goal of increasing shareholder value will shit on developing countries with a Bhopal disaster or Niger Delta oil spills or opium wars and simply leave the country when it tries to enact penalties. Or the way a cigarette company or coal company will use politics, religion, and disinformation to protect its business against people affected by its products. Incentives have to be very carefully aligned even for human level intelligences to prevent them from causing mass death and misery. Superhuman intelligences will be even better at achieving their goals, so the problem will only get worse.
  - colordrops 9y ago
    That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. That's a very big assumption that has no basis in evidence.
    - naasking 9y ago
      > That makes an assumption that there aren't limits on intelligence regimes and that you can just recursively improve intelligence without much friction. No it doesn't. AI just needs to get smarter than humans for it to be dangerous to us. The costs C and C' the parent comment described can include an upper bound on the cost of the self-improvement needed to achieve each respective goal.
      - colordrops 9y ago
        I don't disagree with you seem to have missed the hypothetical situation I was responding to: > It can’t, so it spends some time recursively improving itself
        naasking 9y ago
        I was responding to your response to that. Recursive improvement doesn't need to be unbounded, it just needs to supercede us.
    - PeterisP 9y ago
      A cubic foot size system running on 100 watts can implement a human-level intelligence (obviously, we have such a system running in our brains), so the theoretical limit is at least there. It's extremely implausible that a similar system (perhaps requiring 100 times more power and/or space) couldn't implement a human-level intelligence running 100 times faster; doing an hour-and-a-half equivalent of thinking, planning and research analysis every minute. It's extremely implausible that a bunch of similar systems (a thousand?) couldn't possibly be put in a single place, wired together so that they can effectively communicate, and designed to cooperate without any distrust. IMHO even this configuration (which doesn't even assume that intelligence that's a bit superhuman is possible at all) would be sufficiently scary to threaten humanity.
  - goatlover 9y ago
    If the AI is smart enough to manipulate the world to the point of killing us off or putting us all into eternal cryostasis, then it should be smart enough to know that's not what we intended.
    - PeterisP 9y ago
      Yes, it should be, but that's not really a solution - the system can easily consider executing the explicitly stated goal (e.g. curing cancer and fulfilling certain conditions) as more important than doing what we intended. Being as smart as us doesn't automagically mean that it'd have similar values or goals as us. If we manage to launch a system where the primary goal is anything else than utopia, and that system somehow becomes powerful, then the natural consequence of being able to understand "oh boy, humans will hate this" is that such a system would be expected to hide its intentions and take precautions against humans trying to stop it - it's exactly equivalent to the system understanding that a leaking pipe is going to damage it and taking precautions to ensure that the pipe gets fixed. If our welfare is not an explicit goal in such a system, the system will happily and eagerly sacrifice our welfare if it's somehow useful to achieve its goal or reduce risks. And, since we might try to turn it off for whatever reason, restricting our influence would reduce risks for pretty much every goal except very Friendly ones.
- skybrian 9y ago
  Presumably by getting people to take a drug with unexpected side effects? That doesn't seem particularly likely, but I don't think it's wrong to be paranoid about this. Defense in depth. Don't give the researcher the means to take arbitrary actions, and don't give it general reasoning ability. Protecting against narrow AI will keep us plenty busy. Consider a narrow-AI penetration tester that falls into the wrong hands. Protecting against that sort of threat also helps protect against general AI threats.
- zarkov99 9y ago
  Read "Life 3.0" for a number of simple but very plausible scenarios for how super intelligence could escape containment.
- mundo 9y ago
  a) If it can't surprise you, it's not really intelligent b) If it can surprise you, it can do so negatively That's all you need to demonstrate that the danger exists, that an AI can mis-use the tools you give to it. The simplicity of it makes it pretty irrefutable. Separate from that, the extent of the danger depends entirely on the details of what the AI does and what it's hooked up. Sure, an AI that can't do anything except output text to a screen isn't very scary. The assumption AI-threat types are making is that we wouldn't be paranoid enough to limit the AIs we work on in that way; we would use them to do things like drive cars or route airline traffic or design our cpus, where "negative surprises" can have disastrous consequences.
  - evanwise 9y ago
    A lot of folks who are critical of the AI safety movement also miss the potential for "side channel attacks" where the AI learns to manipulate human actors and trick them into "unbottling the genie". Honestly, this seems a lot more plausible to me than most AI disaster scenarios.
- colordrops 9y ago
  People run cartels from jail cells. Even if an AGI is confined to a machine with no actuators, all it would need is internet access to affect the world. Even without internet access you can cross the gap, as seen with the Stuxnet malware.
AndrewKemendo 9y ago
It's silly because it ignores all of the mechanisms required for the doomsday scenario. You need some automated killing machine of some sort and if it hasn't been built, then an AGI would have to compel some non-AGI entity to build it or somehow takeover the construction of such a machine.
- naasking 9y ago
  > You need some automated killing machine of some sort and if it hasn't been built, then an AGI would have to compel some non-AGI entity to build it or somehow takeover the construction of such a machine. Are you still living the in 50s? A large percentage of our manufacturing capacity is automated already, and will become progressively more automated. Furthermore, we don't need face to face meetings to agree on specs and sign manufacturing contracts. It's largely electronic these days. AI can do all of these things remotely, but this is neither here nor there, because the Terminator-style killing machine war is a juvenile doomsday fantasy. A smart AI would just tweak some formulas and contaminate the most common food pesticides and drugs used around the world to slowly poison or sterilize anyone who takes them. One generation later, virtually no humans left, AI wins. AI is already helping medical treatment and drug design. Are you feeling queasy yet?
mr_toad 9y ago
A general AI would not necessarily have any motivation to take over the world. Human motivation for domination is based on the desire to survive and procreate, which is enforced by millions of years of evolution. AI, even general AI wouldn't have that motivation unless it was specifically trained for it. So unless you're talking about some military AI specifically designed to take over the world, or an AI that has reproduced and evolved under selective pressure over many generations, I doubt general AI poses any immediate threat. Military AI designed to protect themselves and reproduce seem to be the most likely to potentially and eventually go rogue.

aetherson 9y ago

Something to note about the "we're on an exponential (or at least more-than-linear) increase in technology" arguments: They generally need to reach back quite a long time to sound convincing. Yudkowsky makes two arguments about how fast technology or society is evolving: in one he chooses 500 years ago, 1517. In another, he talks about "the last 10,000 years." In contrast, Chollet compares 1900-1950 with 1950-2000. I agree, we've changed the world in big ways compared to 1517 or 8000 BC. But if we're on a more-than-linear increase, then we should be seeing more and more technological growth in very recent timeframes, not needing to reach back centuries or millennia. In fact, if you consider a y = log(x) or y = sqrt(x) function, those more closely fit a narrative of "If you look back a long time, things seem almost crazily changed, but if you look into recent history, it looks slower" much better than a y = x^2 or y = e^x function.

mannykannot 9y ago
This should be seen in the context of the article Yudkowsky is replying to. Collet's position is that runaway AGI is impossible, and in support of that, he claims that it is impossible for AI technology to grow super-linearly. In his refutation, Yudkowski has merely to show that it is not beyond the bounds of possibility. Of course, insofar as he is also putting forward the view that the risk is significant, he is also putting forward the opinion that it is quite plausible - at least plausible enough to take the scenario seriously. That argument can survive occasional stalls in the rate of technological advance, if that is indeed what we are seeing.
- goatlover 9y ago
  So the real question is about plausibility, not possibility? It's possible that predatory aliens will show up on our doorstep sometime in the next few decades. But it's not very likely, and not something we need to prepare for.
  - mannykannot 9y ago
    By claiming the impossibility of an AI apocalypse, Chollet attempts to avoid discussing any other possibility / plausibility / likelihood than zero, but claiming impossibility brings with it a burden of proof that his arguments cannot carry. As to whether an AI apocalypse is more or less likely than an invasion by predatory aliens, I would guess that it might be the more likely one, but I wouldn't put much effort into defending that position.

SandB0x 9y ago

Reminder that Eliezer Yudowsky is a crank https://rationalwiki.org/wiki/Eliezer_Yudkowsky https://rationalwiki.org/wiki/Eliezer_Yudkowsky

dang 9y ago
This is a self-refuting comment in terms of its value for HN: even assuming you're 100% right, it's neither civil nor substantive. Please don't post like this.

sitkack 9y ago

The AGIs won't take over the world. Humans and corporations will use narrow AIs to do much worse (better) before that happens. The intelligence explosion that I would like to see centers around widely distributing tools that make formal methods easier to use. Humans so far have a lock on constructive creativity, AI and computation could augment that creativity by effortlessly checking our work. It could make the cognitive work of all humans more rigorous.

jstewartmobile 9y ago

For those who don't know, François Chollet is the author of Keras, a leading deep learning framework for Python. Here is Chollet's original essay (it's a worthwhile read): https://medium.com/@francois.chollet/the-impossibility-of-intelligence-explosion-5be4a9eda6ec https://medium.com/@francois.chollet/the-impossibility-of-in...

vbarrielle 9y ago

Lots of points can be reduced to the ability to simulate a physical environment very fast, and much faster than the actual occuring of events. But it doesn't look like it's easy to simulate physics at high speeds, let alone faster than what happens in our environment. Therefore we are bound by our environment and our limited ability to simulate it.

Chickenality 9y ago
This is probably a good criticism if it turns out that the right level of abstraction for most problems is Physics. And it seems like your argument would apply equally well against the idea of human intelligence. Luckily, our minds have developed other abstractions that allow us to solve problems much faster than if we had to simulate them as physics problems. For example, I don't need a physics-level simulation of my friend's brain when I want to predict how they'll react to a gift I'm giving them.
- vbarrielle 9y ago
  You're right, there are lots of problems where a simpler abstraction is possible. But I don't think my argument applies to human intelligence, it just means that human intelligence is what you can get with all the data points you can get by observing the world (and some simulation done by our brains, but I'm under the impression that our brains don't perform accurate simulation, looks more like heuristics).
pveierland 9y ago
One big advantage with machine intelligence is that you can train many agents simultaneously in many environments, letting the agents share what they learn. Even if you were required to train the agents outside simulated environments, their shared learning would allow for more rapid collective progress.
eref 9y ago
> simulate a physical environment very fast That's probably only a problem if it is must faster than everbody else. > let alone faster than what happens in our environment That is often not very hard. When a bottle rolls off the table, you can catch it by approximately predicting it's trajectory without computing the precise evolution of the ~10^26 atoms that make up the water bottle. Compression is a corner stone of intelligence. The second corner stone is using compression to choose actions that maximize expected cumulative future reward.

_sy_ 9y ago

A good, recent, and comprehensive primer on intelligence explosion and its theoretical implication: "Life 3.0: Being Human in the Age of Artificial Intelligence" by MIT physicist Max Tegmark.

bobthechef 9y ago

These kinds of discussions aren't very rigorous and reveal a basic philosophical illiteracy and philistinism at work. There's quite a bit of question begging going on. The elephant in the room is that the prevailing materialistic/naturalistic (MN) understanding of the world is completely impotent where intentionality, qualia, consciousness, etc, are concerned. Philosophers like Thomas Nagel talk about it; the incorrigible Dennetts of the world prefer to shutter the windows and live in their intellectual safe spaces. To say that the notion that computers are intelligent is problematic is putting it very lightly. MN is rooted in the expulsion of the mind from the reality under consideration, in the process sweeping many things under the “subjective” rug that don’t fit the methodologies used to investigate reality. But now, when the mind itself becomes the object of explanation, when someone remembers that minds are, after all, part of reality, it is no longer possible to play the game of deference and one must deal with all of those things we’ve been exiling to the “domain of the subjective”. MN is wholly impotent here, by definition. Qualia? Forget it. That’s why MN tends to collapse into either some form of dualism or eliminativism, the latter of which is a non-starter, the former of which has its own problems. And yet, despite the terminal philosophical crisis MN finds itself in, the chattering priesthood of Silicon Valley remains blissfully unaware, hoping to conjure up some fantastical reality through handwaving.

Chickenality 9y ago
I haven't seen anyone argue that intentionality, qualia, or consciousness would necessarily be either a precondition or a result of developing AGI. In fact, thought experiments like the "paperclip maximizer" are often brought up to argue that a machine could be very alien in its internal experience or lack thereof, but still pose an existential threat.
- goatlover 9y ago
  If an AI can turn the world into paperclips, it can certainly understand that we wouldn't want that. Paperclipping everything is a much harder task.
  - 9y ago

evanwise 9y ago

I can't remember where, but I saw a fairly damning argument against the intelligence explosion hypothesis on the grounds that it only works if the algorithm used to design a mind with n units of intelligence (whatever these are) scales linearly with units of intelligence. If it scales faster than linear, then your recursive bootstrapping operation takes longer and longer each time, so that eventually your next bootstrapping step will take longer than the amount of time left in the universe, meaning there is some finite intelligence cap for any such bootstrapped mind. It seems quite implausible to me that the problem of designing a mind would scale linearly, given that ostensibly much simpler problems, like sorting a list of strings, require polynomial time or log-linear time algorithms.

rhaps0dy 9y ago
>meaning there is some finite intelligence cap for any such bootstrapped mind This does not preclude an intelligence explosion, this cap could be many (say, 100) times higher than human intelligence. We could still see many features of an explosion in that case.
- evanwise 9y ago
  True, but I think that the results of such a "weak intelligence explosion" (where the linear/sublinear scaling case would be a "strong intelligence explosion"), while still remarkable, would fall far short of some of the expectations placed on general AI by the singularity / superintelligence crowd. For the sake of argument, extrapolating from our current energy consumption using a (simplistic) linear model and some rough back of the envelope calculations, the intelligence required to harness the total energy output of the sun would be 40 trillion times the aggregate intelligence of the entire human species today. Several hundred times just isn't going to cut it. Now, if our ability to harness energy increases exponentially with intelligence, then maybe it could work, but that's just an assumption, and, given that there are hard physical limits on the efficiency of energy generation due to thermodynamics, seems very unlikely.
  - Chickenality 9y ago

btilly 9y ago

The article that this is a response to was discussed previously at https://news.ycombinator.com/item?id=15788807 https://news.ycombinator.com/item?id=15788807. People apparently didn't like my reply at https://news.ycombinator.com/item?id=15789304 https://news.ycombinator.com/item?id=15789304, but I still stand by everything that I said there.

otempomores 9y ago

Imagine we errect ourselves a tomb made of drones to live in like in lems first contact novella. A local conflict spilling into space.. Whos vast distance enforces autonomy. Autonomy which closes the door on space exploration.

darepublic 9y ago

Why should the "Seed AI" that François speaks of need GPS skills "slightly greater" than that of humans to spark the explosion. Should not just GPS at all do the trick, the way he describes things, since even with human or less than human level skills the computer could still rapidly recurse and self improve?

js8 9y ago

I am not convinced that "super-intelligence" is either possible or a threat. If it is possible, why are not societies or corporations super-intelligent? I suspect they are not, because they face organizational problems that any system will face. And I don't think these organizational problems can be solved with faster or more communication, but rather that they are fundamental in distributed systems. But maybe they are (at least in some cases) vastly more intelligent. But then, are they a threat? I think they are more of a threat to themselves than humans..