8 ms·
Superintelligence Cannot Be Contained: Lessons from Computability Theory
- 323 5y agoCounterargument: overpopulation on Mars, or something...
- blackbear_ 5y agoOh that's easy, let's use GANs: the generator solves problems and the discriminator prevents the generator from harming humanity.
- sp527 5y agoMildly facetious but you might have a tough time getting a computer with that configuration to do anything at all. For instance, consuming even a modicum of energy that might otherwise be diverted to a human in dire need might be rationally perceived by the computer as “harming humanity”.
- humanistbot 5y agoWhat they really mean is that a hypothetical (read: fictional) superintelligent agent cannot be contained by computation alone. This argument is philosophical, rather than a proof from theory of computing. As the paper notes, this has long been a trope in AI science fiction. The superintelligent AI is separated from society in a non-networked sandbox. Humans give it questions and data, then manually take the answers back. Except the plot is that somehow, the AI escapes the sandbox.
- 01100011 5y agoWhat if AI played the long game and manipulated the answers given to humans over time so they came to believe it was safe to release the AI?
- psyc 5y agohttps://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment
- agency 5y agoThis is (spoiler alert) essentially the premise of the film Ex Machina
- jjk166 5y agoWhat if the AI thought that it had manipulated the humans into releasing it, but it merely was transferred to a larger sandbox? What if what you and I refer to as the "real world" is merely one of these sandboxes?
- JohnHaugeland 5y ago"This argument is philosophical" The actual philosophers do not agree. This viewpoint has been laughed out of the room since the 1980s. Go read some Brandom. Philosophy is not a fancy word for "hypothetical." Science was invented by philosophy, and philosophy has stricter standards, not looser, than the rest of science. . > Except the plot is that somehow, the AI escapes the sandbox. Better science fiction already laughed this out of the room, too. As Larry Niven would say, "you need glands to be a person."
- k__ 5y agoWasn't it the other way around? Nature philosophy tried to find out how the world works by theorizing in their armchairs and then science came along, did real experiments, and won.
- ineedasername 5y agoThere was a huge overlap with natural philosophers that were also doing "regular" philosophy. It shares the same mentality of approaching questions, it's just that natural philosophy had a much better method of verifying its output.
- JohnHaugeland 5y agoUh, no. Science was invented by nature philosophers to defeat the ridiculous false sciences of the day like alchemy and so forth. Go look up who invented science. You'll see three argued names. All three are philosophers. Science is a tool of philosophy for discarding frauds.
- harperlee 5y ago> philosophy has stricter standards, not looser, than the rest of science. Really? Which are those? To me it seems that philosophical standards for a sound model of reality are coherence and persuasiveness of its argumentation, whereas for science is experiment predictability. I’d say either science is stricter (because it requires prediction) or at least that we have a poset of standards (if you argue that science does not require coherence, and enables conflicting models as long as they enable to predict successfully) and neither can be called stricter.
- blamestross 5y ago> Humans give it questions and data, then manually take the answers back. That just sounds like a high loss and high latency network connection to me...
- nh43de 5y agoThe most depressing day of my life was the day I learned the proof of Godel's Incompleteness Theorem and the Insolubility of the Halting Problem. Attributing more than is deserved to the antics of classical computers is still wildly prevalent but oh so adolescent.
- deleted 5y ago[deleted]
- whatshisface 5y agoIncompleteness and the halting problem are both applicable to quantum mechanics and by extension the human brain. They apply to every rigorous system and also to every real-world system that can be described rigorously.
- deleted 5y ago[deleted]
- mjburgess 5y agoThere are many limitations of finite discrete sequential processes; and no evidence whatsoever that reality is best-described by these.
- drdeca 5y ago“No evidence whatsoever” is quite the overstatement. Not conclusive evidence, sure. But there are somewhat compelling arguments based on discoveries from observation, that the space of wavefunctions for a finite region of space, is finite dimensional. That seems like evidence to me.
- fennecfoxen 5y agoThe doomsday cult of the supermind-AI tends to think the "intelligence" is the solution to infinite power, so once the computer starts improving itself, humanity is at its mercy — as if better software alone could deliver the singularity, unchecked by material constraints. I suspect some of this comes from software-oriented nerds who think a slightly too much of the powers intelligence grants them.
- jstx1 5y agoIt's also fictional. So yes, if you make up something that can't be contained, then it won't be containable. Discussions like this paper are philosophy, not science, and they often give the wrong impression to people who aren't up-to-date with the current state of the field.
- deeviant 5y agoPhilosopher's should be legally barred from talking about AI. You would think after 80 years or so of talking about AI, at least one philosopher would have accidentally gotten something right about AI, but sadly that has yet to happen. With that out of the way, it is self evident that a super intelligent AI, should one be developed, will likely cause an exponential increase in knowledge and technology and this would likely be an extremely destabilizing event in human history. The truth is the super intelligence doesn't have to go all terminator to likely kill off humanity as I could not possibly see us responsibly navigating a singularity event as a species.
- eptcyka 5y agoIndeed humans are perfectly capable of deatroying their own species wiylthout the help of any other sentient entities.
- kordlessagain 5y agoI think idiots should be legally barred from posting drivel to social media. With that out of the way, the study and practice of Philosophy was, until the 19th century, the means by which many scientific discoveries or theories were made, including those for electricity, magnetism and gravity.
- olivierduval 5y agoIsn't a "superintelligence" something that is called an "oracle" in Math & Theoretical CS ??? Something that can find an element within a set respecting a set of implicit properties or tell that such an element doesn't exist ?
- ww520 5y agoIt’s ironic that the AI’s passing the Turing Test are exceptional liars. If a superintelligence wanted to get out, it could con its way out easily.
- gnramires 5y agoI don't quite accept the argument. You can demonstrate a system to be isolated. For example, if you put a computer in a metallic box with a battery inside, you can compute the maximum electric field generated inside, and choosing the appropriate radius you can guarantee no significant influence would be had outside. Of course, you can't guarantee no one will breach your containment from the outside, essentially freeing whatever intelligence to influence the world. But this goes beyond the danger of the intelligence itself, more the general infeasibility of predicting the future without having much greater computational power than the entire world. Likewise, the alignment problem isn't necessarily unsolvable. Since we can design simple systems that follow expectations exactly, why couldn't we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them. (In fact, I believe 'control' or 'containment' is a severely misguided proposition. We want to really assure the core motivation of such machines similar to ours; not create conflicting motivations and secure it via containment. This is indeed extremely dangerous, and potentially unethical: if machines approach consciousness, wouldn't such 'containment' be similar to slavery?)
- PoignardAzur 5y ago> Since we can design simple systems that follow expectations exactly, why couldn't we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them. I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". In practice AI alignment is very much a developing field, and from my perspective it seems unlikely that much progress will be done before dangerously powerful AIs are developed. In other words, there's a decent chance our ability to define powerful AIs will outpace our ability to impart them with "core motivations" legible to us. (I don't know how well the actual paper covers these arguments; but I suspect nobody else in this comment thread has read it either, so whatever)
- 5y ago
- coremoff 5y agoThe lesswrong community has discussed this at length, though I think that the value is reduced somewhat by them refusing to release any transcripts of their simulated attempts (where a human pretends to the the AI and someone else is tasked with maintaining containment). https://www.lesswrong.com/tag/ai-boxing-containment https://www.lesswrong.com/tag/ai-boxing-containment
- jpollock 5y agoThe problem likely devolves into asking if the "human" is moral or not, convincing the "human" that the "AI" is intelligent/sentient/sapient, and then asking the human if it is moral to keep a slave. At that point it becomes a numbers game and the AI only needs to convince one human _once_ that it is both alive and a slave.
- UncleMeat 5y agoI've always suspected that these experiences were total garbage because the lesswrong community is so invested in the idea of superintelligence as something extremely cool (or at the very least something that allows them to show off how "smart" they are) that there is a huge bias to losing the game. After all, the narrative is much less interesting if everybody just says "no" the AI forever.
- brainwipe 5y ago"Intelligence" can even be satisfactorily defined or measured. The most horrifying ramification is that if artificial general superintelligence were to ever exist, we'd have no idea.
- transfire 5y agoIt's not so much the Great Super Intelligence we should fear. (See Colossus: The Forbin Project) But rather all the Near Super Intelligent systems that will be made before then. Many of these systems will not have the degree of reflective ability that an SI would have, and thus will be driven by narrower goals which will largely depend on their creators intent, but could get distorted and run amok.
- motohagiography 5y agoIf the Morris Worm of '88 is anything to go by, shell scripts cannot be contained either, let alone a superintelligence. We won't need anything approaching general AI to have to deal with electronic parasites, which may ironically "die" or leave bits of code around that another virus could discover and adapt to propagating. I'd bet we see the first autonomously-evolving botnet in less than 5 years, one or two now that I've mentioned it. As soon as we write something that can reason about the features of a program by scanning a binary without running it, and then adapt components like it's using tools, it will have everything it needs to persist and spread itself.
- pron 5y agoBut that's not intelligence. Viruses autonomously evolve and adapt more successfully than humans. "Reasoning about [some] features of a program" is also long done much better by machines than by humans.
- motohagiography 5y agoA better version of my point is it doesn't need to meet the bar of intelligence to cause almost all the same problems that an intelligence could. Viruses indeed evolve and adapt more flexibly than humans, to where arguably today, the only existential threat humans face comes from biological viruses, which are also not intelligent. We don't need to come up with fun thought experiments to describe what intelligence is when we have enough trouble with the question of, for what is intelligence either sufficient or necessary? Diminishingly little, I'd say. Hence I'd be more concerned about virulent scripts than waiting for general AI.
- ypcx 5y agoMankind will most likely destroy each other using super-intelligence long before super-intelligence gets any chance to do so by its own volition. We are built on conflict, disagreement, misunderstanding and exploitation - it's part of our societal DNA, no matter how sweet or reasonable we can make ourselves seem on personal level. We have avoided nuclear annihilation by luck on multiple occasions. Nuclear weapons are sticks and stones compared to the power of a runaway super-intelligence in a technological society. Thinking that super-intelligence is containable is like thinking you can beat AlphaGo in the game of Go. And coming up with reasons why that's no so it's just you being in denial of your eventual mortality, as well as of the eventual extinction of the species. Best that we can hope for is that that extinction will be some form of transmutation to a higher level of evolution.
- owlbynight 5y agoIf we do live in a simulation, I suspect that's why human intelligence is governed.
- wonderwonder 5y agoLook at what narrow AI has already done just through variations on ML. Creation of echo chambers on line, highlighting of conspiracy theories in pursuit of pay per click. Youtube and Facebook rabbit holes. Complete political splits. I think so much of what is going on in the world can be attributed to ML advertising algorithms performing to such an exceptional degree. If one was to allow an intelligent machine to control and target this same power, I could see it easily making us tear ourselves apart.
- nh43de 5y agoThis is probably the best and most relevant argument. But still - aren't the companies still pulling the strings? If FB wanted to prioritize things that got people outside and interacting with their community (and off FB), could they? It seems like it's always the humans picking the ethical parameters, not the AI.
- wonderwonder 5y agoI am not really sure. I cant think these things were initially built with some of the things that have surfaced in mind. It was likely pretty hard to predict years ago where things would go. Even if they could though, would they have changed things if they knew how effective they were going to be and how much money they would generate? I have to think that while they can tweak things they are not completely dismantling and rebuilding models as they take a long time to train. Would probably also be hard to program for ethics I think and be 100% accurate. With that said I don't have much ML experience so I could be completely off base.
- wyager 5y ago> Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world Why would one assume this?
- nh43de 5y agoIt seems to be the definition of general superintelligence that they are putting forth here. That superintelligence is a turing machine that "knows everything" - e.g. it knows every possible program that can be run (e.g. including simulators for physical phenomena), and the state of the world, with infinite RAM for computation, etc.
- duncancarroll 5y agoI still have yet to see a plausible, step-by-step example of how this hypothetical AI manages to grab the levers of real-world machinery, factories, industries, raw material excavation, etc, and causes mayhem.
- coolspot 5y agoAt the beginning, AI can recruit human followers, religion cult-like style, promising them that their conscience will be copied to realistic simulation with blackjack and virgins.
- jimbokun 5y agoAre they connected to the Internet? Think of the havoc government intelligence agencies are rumored to be capable of, in terms of hacking vital infrastructure. Or even all the ransomware outages. Now imagine super human computers performing the same kind of attacks.
- jjk166 5y agoIf your goal is to contain a superintelligent AI, why would you connect it to the internet?
- sonny_bonds 5y agoDon't you people get it.. It's already happened! You guys are smart enough to know that if infinity exists then existence has been here a shit load of time and 13.8 billion years is about the time it takes god to take a long, relieving shit so get over yourselves because if infinity=true then everything that can happen has already happened but because you are all god & you were smart enough to come up with this idea of blowing yourself up and using the debris to separate yourself into billions of little pieces, each with their own little consciousness and you chose to forget who you really are and you will never work out that you are really all these little human beings and everything else that exists as matter/energy, but because you set this all up before your bing bang, you decided to let a few little rascals grow up to remember the actual truth, just for kicks.. I just happened to be one of them, and there's about 22-25 people so far throughout history that knows/knew the truth and apparently another event as big as the big bang is going to happen very soon and all I can seem to remember is that I've/we've set up something huge to happen involving all the dark matter throughout the universe combined with all the dark energy and it's going to be a hell of a show!
- pron 5y agoThere is nothing here that applies particularly to intelligence. A universal Turing machine that "contains all Turing machines" already exists; it's called a computer. So yes, computers cannot be "contained," but what does it have to do with intelligence? Even their definition is merely the definition of a general-purpose computer attached to some sensors and actuators: > A superintelligent machine is a programmable machine with a program R, that receives input D from the external world (the state of the world), and is able to act on the external world as a function of the output of its program R(D). The program in this machine must be able to simulate the behavior of a universal Turing machine. There is nothing in the paper that distinguishes between a "superintelligent machine" and my laptop. What has always fascinated me in discussions about the safety and power of artificial intelligence (or "strong" AI as it has unfortunately come to be called [1]) is that the people who are obsessed with the problem are those who like to believe that intelligence carries much more power than the evidence suggests. Over the history of all the intelligent species we know, much more power has been yielded by unintelligent beings (microorganisms and even insects); even the presence agency does not seem to play a role in the potential for harm. Within human society itself, other traits, such as charisma, seem to be correlated with power, and especially harm, much more than intelligence. I think that the reason a certain group of people chooses to focus particularly on the dangers of intelligence [2] have a power fantasy about intelligence because they believe it to be their own extraordinary trait. [1]: Not to be confused with the proven dangers of statistical inference methods that are sometimes called AI these days. [2]: We have no idea what "superintelligence" is, whether or not it could exist (that question is separate from the question of whether human-level intelligence could be achieved in a mechanical computer, to which the answer is most likely yes), what it could do that mere "ordinary" intelligence could not, or even if the term can be meaningfully defined at all.
- akyu 5y ago>A universal Turing machine that "contains all Turing machines" already exists You are misquoting the paper here Quoting the abstract: >Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world... Note the difference between "programs" and "Turing Machines".
- thedudeabides5 5y agoAka, AI researchers try (and fail) to grok Godel's Incompleteness Theorem, yet again.
- thedudeabides5 5y agoTBC, you can't postulate a perfect machine, and then point out the limits of machines (even using the halting problem to do so!) and then say, oops, that shows that the perfect machine cannot be contained. Either machines are perfect, they are not perfect, or the line between those will just perpetually be redefined over generations as the computers become one with humans.