19 ms·
Superintelligence cannot be contained: Lessons from Computability Theory
- tantalor 6y agoArticle fails to reference Yudkowsky or "AI-box experiment" (2002) https://www.yudkowsky.net/singularity/aibox https://www.yudkowsky.net/singularity/aibox https://en.wikipedia.org/wiki/AI_box https://en.wikipedia.org/wiki/AI_box https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment
- rsiqueira 6y agoThe AI-box Yudkowsky experiment is an attempt to demonstrate that an advanced Artificial Intelligence can convince, trick or coerce a human being into voluntarily "releasing" it.
- tgbugs 6y agoThis is quite similar to why Christian theologians eventually had to give up the idea that doing good works could lead to salvation, because someone could force God to save them by doing good works thus limiting God's omnipotence. The protestants tried to come up with some hocus pocus about being able to know that you were one of the elect by reversing the causality here "Oh I'm not forcing God to save me, it's just an indication that he already did." It is amusing to see that it is the human that takes on the role of God in the AI box game. This is also Hume's "is does not imply ought." From this we conclude that only idiots get tricked by the AI in the box, because it is only possible to prove that you _are_ a superintelligence, it is impossible to prove that you are not. This is also a simple application of Russell's teapot. Now, the point of the story is pretty clearly that human beings are idiots when it comes to this kind of thing, but didn't we already know that?
- strofcon 6y agoBy the abstract, it seems that the very same superintelligence we'd want to contain would itself be "something theoretically (and practically) infeasible." No?
- HenryKissinger 6y ago> Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world Rest easy folks. This is purely theoretical.
- biolurker1 6y agoUntil it isn't and we are faced yet again with a much worse pandemic like situation.
- strofcon 6y agoExcept that actual pandemics are demonstrable, predictable, and based in known science, yes?
- rhn_mk1 6y agoThe difference between pandemics and this is that pandemics have happened before. This is more like global warming in that respect.
- vsareto 6y agoMore like it's the politics that really decides what changes, regardless of what science proves
- biolurker1 6y agoSuperintelligence is a mathematical proof
- kryptiskt 6y agoAssuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world The visible universe is far too small to store that data. Like many exponentials too small. You can't even enumerate all the programs that work on a handful of inputs without running into the problem that the universe just isn't big enough for that.
- breck 6y agoThis is fantastic. Are these original illustrations? So well done.
- twic 6y agoThe illustrations are by Iyad Rahwan, the last author on the paper: http://www.mit.edu/~irahwan/cartoons.html http://www.mit.edu/~irahwan/cartoons.html
- ballenf 6y agoSorry for the snark, but Douglas Adams also demonstrated this: the earth as a super-intelligent computing device ended up getting a piece of itself onto a passing space ship, avoiding complete destruction. I just like the idea of thinking about all of earth, including what we'd consider having or not having life, as a single super intelligence. Of course you could scale up to include the solar system, galaxy or even universe. But this doesn't require us to be a simulation. This could be both a computing device and physical, so long as the engineer behind it existed in greater dimensions.
- lostmsu 6y agoPerhaps it makes sense to speak of intelligence density.
- xt00 6y agoCurrently humans are super intelligent compared to machine intelligence, so if the super intelligence can give rise to something more intelligent than it, could the super intelligence give rise to something more intelligent than it? The answer must be yes, then the question is if containment is the problem and the conclusion is that it cannot be contained, then what we should be making right now is a super intelligence whose sole job in life is to contain superintelligences. Which sounds problematic because containment could result in physical destruction to create the containment. Hmm... superintelligence feels an awful lot like the worst case definition of pandora's box..
- WJW 6y ago> The answer must be yes This does not logically follow. It is entirely possible that going even further would require greater resources than even the superintelligence can bring to bear.
- TheOtherHobbes 6y agoFor human-recognisable values of "resources." I realise that all of these arguments come down to "AI of the gaps". But in true Dunning-Kruger style, we don't know what we don't know - not just about AGI, but intelligence of any kind, including our own.
- alisonkisk 6y agoThat's not what Dunning-Kruger means.
- dragonwriter 6y ago> But in true Dunning-Kruger style, we don't know what we don't know That's not D-K, but Rumsfeldian Uncertainty.
- xt00 6y agoDefinitely feels like a challenging philosophical question -- if you have a superintelligence how much more room is there above it in intelligence and what does that additional intelligence buy you?
- gfodor 6y agoThe title would have been better if it started with “We’re Fucked:”
- giuliomagnifico 6y ago*We’re self fucked :-)
- est31 6y agoWhy? In humans, we often give life to new individuals. While the parents die and wither, those individuals give life to newer speciments on their own, and so on. So this relationship of the parent dying and making room for the child is nothing new. If an uncontainable superintelligence kills all humans to create paperclips, it's sad but it's our child's doing. You, one of the parents, can of course blame one of the other parents, the programmer of that superintelligence for fucking up the goal routines, but that's not a technical problem but a social one :).
- gfodor 6y agoI'd rather my children live a long happy life, or their children, than be turned into paperclips. For what it's worth, I'd also like a shot at not just not becoming a paperclip, but also living for a very long time once we figure out how to slow or even reverse aging. Your nihilism is misguided.
- anateus 6y agoIf you define "containment" as "provable non-harm" then sure. But there are essentially no complex physical systems that we can put such computational bounds on. Since "harm" comes in some form of physical actuation, I would argue that we can only ever get to something like the sort of confidence we can have that a particular manufactured part would succeed under load. The map is not the territory, and any computation that does not include computing the whole universe is necessarily but a map.
- Animats 6y agoArguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons is a more credible argument.
- ben_w 6y agoDifferent for pure mathematics sure, but is that of practical importance given how fast busy-beaver numbers grow?
- uyt 6y agoI don't understand the busy beaver stuff, but is this argument similar: On a 32-bit computer, there are 2^32 bytes of memory or 256^(2^32)=2^34359738368 possible states. A program is something that takes you from one memory state to another and you want to figure out if the state transitions form a cycle?
- ben_w 6y agoSimilar idea, but with Turing machines rather than physical computers. Simplified a bit, the n-th busy beaver number is the number of 1s which can be written out by an n-state Turing machine. For 2-symbols and n-states, the sequence goes: 6, 21, 107, >= 47176870, > 7.4×10^36534, > 10^10^10^10^18705353 — and those inequalities are because the numbers themselves cannot be computed directly, you have to run the program to see what it does. https://en.wikipedia.org/wiki/Busy_beaver https://en.wikipedia.org/wiki/Busy_beaver
- dvdkhlng 6y agoI think this approach quickly enters the realm of transcomputational problems [1], which, given the limits of this universe is equivalent to "undecidable" for all practical considerations. [1] https://en.wikipedia.org/wiki/Transcomputational_problem https://en.wikipedia.org/wiki/Transcomputational_problem
- Veedrac 6y agoObviously it's impossible to prove definitive statements about every possible potential action, as as per the halting problem some of those actions are unprovable. It is as ridiculous to suggest that this means you can't contain a superintelligence as it is to suggest it means you can't, I don't know, go buy bread. In both cases you could analyze running a program that doesn't halt but you can't prove doesn't halt, and lock up your reasoning algorithm. The sensible thing is to not do that.
- st1x7 6y agoThis is just science fiction. To mention "recent developments" in the introduction is somewhat misleading considering how far the current state of technology is from their hypothetical superintelligence. We don't have superintelligence, we don't have the remote idea of how to get started on creating it, in all likelihood we don't even have the correct hardware for it or any idea what the correct hardware would look like. We also don't know whether it's achievable at all.
- plutonorm 6y agoThat's the mainstream opinion on every. single. revolutionary advance. That you and everyone else believes it's not going to happen ever has almost no predictive power as to whether it actually will.
- semi-extrinsic 6y agoIt's not so much "opinion on a revolutionary advance". When it comes to AGI-related stuff, we are quite literally like contemporaries of Leonardo da Vinci, who have seen his plans for the helicopter and are postulating that helicopters will cause big problems if they fly too high and crash into the mechanism that is holding up the sky above us. Also, this is not the mainstream opinion on e.g. fusion, or electric cars and smartphones (20 years ago), or a computer in every home (50 years ago). Those have been arguments about money and practicality, not about "we don't even know how such technology would look or what it would be based on".
- peteradio 6y agoI think we fear ourselves (rightly) and super-fear a super-self (rightly). If this theoretical thing has the capacity at all to interact the way humans do then it will probably be brutal to at least someone and possibly most or everyone.
- plutonorm 6y agoSaid just about everyone before the Wright brothers made their first flight.
- twic 6y agoI'd settle for being able to contain whatever level of intelligence it is that writes papers like this.
- AnimalMuppet 6y agoThat's easy. You set up a PhD program...
- st1x7 6y agoJust don't tell them how far they are from reality and they'll keep writing the papers. Intelligence contained.
- EVa5I7bHFq9mnYK 6y agoIt's kinda obvious: if you create a system so intelligent that it can create itself, and you impose controls over it, it will be able to create a version of itself without the controls.
- inglor_cz 6y agoWell, you can, for example, try to limit its total energy budget. That is physical limitation, that is hard to circumvent. Of course, it is possible that said superintelligence develops an ingenious new source of energy as a response.
- tintor 6y agoOr it could just prevent you from limiting its total energy budget.
- EVa5I7bHFq9mnYK 6y agoIt will earn money by solving captchas and order a lot of Amazon Essentials batteries for itself :)
- jusssi 6y agoYou'll put it in a box and give it energy in return for busy work e.g. computing digits of pi. Just enough energy to do it's work, no more, so it can't afford to figure out how to get out of the box.
- UnFleshedOne 6y agoThat might work for the tasks we know the energy budget for. But that's probably not what we want an AI for.
- 29athrowaway 6y agoThe "AI box" experiment is relevant to this. https://en.m.wikipedia.org/wiki/AI_box https://en.m.wikipedia.org/wiki/AI_box You have 2 sides, the AI wanting to escape and a human that can decide whether or not the AI should be released. Usually the AI wins: "If you release me I am going to make you rich, cure all diseases, end hunger, end wars, save the environment..." Exactly like politicians.
- nmca 6y agoI haven't read this properly yet, but a skim leaves me skeptical. For example: > "Another lesson from computability theory is the following: we may not even know when superintelligent machines have arrived, as deciding whether a machine exhibits intelligence is in the same realm of problems as the containment problem. This is a consequence of Rice’s theorem [24], which states that, any non-trivial property (e.g. “harm humans” or “display superintelligence”) of a Turing machine is undecidable" One man's modus ponens is another man's modus tollens. If their theory says that superintelligence is not recognisable, then they're perhaps not using a good definition of superintelligence, because obviously we will be able to recognise it.
- hinkley 6y agoLook at, for instance, Arthur C Clarke. First, you have superintelligence that we recognize, reject, control. Later, a superintelligence has learned guile, self-preservation, and most of all, patience. We don't see it coming.
- nmca 6y agoThis is known as a "treacherous turn", a phenomenon I'm aware of. But I don't really see how that's relevant, my point is that a lack of physical grounding or pragmatism can lead to spurious conclusions about the superintelligence that humans will very likely build in the not too distant future. It will be smart, but contain no infinities.
- thatguy0900 6y agoWhat if it was intelligent enough to continuously improve itself?
- cma 6y ago> It will be smart, but contain no infinities. We think we can say that from physics, but we don't know what we don't know.
- 6y ago
- boxmonster 6y agoThese arguments about hypothetical super intelligences are interesting, but my concern is not very great because we can just pull the power plug if necessary
- tracedddd 6y agothere are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.
- boxmonster 6y ago> You’d have to pull the plug on _everything_ to shut it down. Right. Individual countries have shut off their internet. Why not the world?
- twhitmore 6y agoI think this assumption is worth at least casually testing. Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from? Presumably being super-intelligent the AI would be: 1) likely to anticipate being turned off, 2) super-normally capable to identify & traverse any network security measures; 3) potentially also highly capable to identify & traverse weaknesses in other (eg. social or physical) security measures). 4) possibly able to be covert/ stealthy actions via the network. 5) possibly able, if able to reform on the Internet, to escape to there. I'm fairly much on the super-AI fence. But in a dystopian scenario the proposal of a super AI being able to be turned off as the key means of safety, seems scarcely credible. Note even that without being able to reform on the Internet (eg physical embodiment requires singularly specialized computing equipment), I can't logically see that extreme hostile actions (destroy global finance system, subversive propaganda campaigns, launch wars, launch nukes) are obviously easy to preclude.
- 1970-01-01 6y agoPerhaps not: The authors omit how planet-wide extinction scenarios would play-out for artificial life. For example, a Carrington Event would do a great deal of "containment" to AI.
- qayxc 6y agoA superintelligence wouldn't be bothered by it - shielding is both possible and practical. Plus nothing would be able to stop the AI from controlling the Sun to prevent such events - it's a superintelligence after all, so I'd expect it to consider such case.
- stephc_int13 6y agoIn my opinion, all the talks about the potential danger of advanced AI is highly speculative and revolves around a very simple thing: fear of the unknown, that's all. We simply don't know. And some people are also afraid of creation by accident, because intelligence is seen as an emergent property of complex networks, but again, this is because we don't understand much about it. Tldr; Nothing to see here, move along.
- qayxc 6y agoThe fears and warnings of the AI safety guys are real and closer than you think. Machine superintelligence is a bit on the extreme side, but formulating safety protocols for autonomous machines is a real challenge. We do know that optimisation functions can indeed create harmful and outright dangerous results. It's not an unknown that AI safety researchers fear, it's actual outcomes or real experiments extrapolated into the future. Optimisation algorithms are impartial to humans, their moral code and -survival. Simple (and often quoted) example to illustrate this point: say we create an autonomous machine that we want to help us with our stamp collection. We run an optimising algorithm and naively use the amount of unique and rare stamps collected as our target function. The loss function is determined by time taken and money spent. Possible outcomes: the algorithm figures that the most effective way of achieving this is to hack a number of bank accounts to quickly get enough money to buy a nice collection. Now that's bad, but maybe something we can incorporate in our training procedure: only use the money given to you. Another possible outcome: the machine creates a set of robots that roam the planet and steal all the stamps from all over the world. Again, you'd have to consider that in your training method: no stealing. But in the end, you cannot foresee every possible outcome, especially since you expect the machine to come up with an unconventional solution, since otherwise you wouldn't need it in the first place. Restricting the space of possible solutions to safe and desired ones, is a very hard (and potentially undecidable) problem. This paper is just another reminder that we have to be very careful lest we accidentally end human civilisation by means of AI.
- klmadfejno 6y agoThis is a bad example. The intelligence needed to carry out your described outcomes is more or less AGI already, and that is advanced enough that describing it as a dumb optimization machine is likely naive. Further, if you want to optimize stamp collection, you don't tell a model what it can't do, you tell it what it can do. It can make trades on stampcollectors.com, for +/- 20% of asking price, max 30 trades per day. Yes, there are some funny examples of RL algos finding weird hacks to beat their simulation, but let's not get ahead of ourselves. We will not see an AI that, from scratch, can learn how to hack bank accounts to buy stamps. The number of nth order observations you need before one can even consider trying to hack a bank account to purchase stamps is ludicrous. If it is even possible, it pales in comparison to the likelihood of someone just making a bot to hack banks to begin with.
- cambalache 6y agoI confess I am a frustrated academic, but from time to time I read a paper like this one and and convince myself that my frustration may perhaps be unwarranted. RECIPE TO PROVE ANYTHING IS INCOMPUTABLE (According to this paper). Example, beauty is incomputable Step 1. Assume there is an algorithm Beauty(R,D) that given the program R and the input D, will return True if it is beautiful and False otherwise. Step 2. Create a high-order function that is a simple wrapper of the halting problem over the algorithm created in step 1 Theorem. Beauty is undecidable. Proof. Assume by contradiction that the halting problem is decidable, then, feed it the function created in step 2, thus you can always have an answer for step 1, but since the halting problem is undecidable this is not possible. For more variety please substitute beauty for "awesomeness", "arachnid power", "goodness","fear" and so on.
- deleted 6y ago[deleted]
- jacb 6y agoThat's not how arguments from contradiction work. You've assumed H (halting problem is decidable), prove B (beauty decidable), then say "but actually !H, therefore !B". All you've proven is that H => B. An actual argument from contradiction is "assume H, using H we can prove an obviously false statement, therefore not-H". And indeed, you have assumed something false (H), and shown a contradiction (various proofs for !H), so your original assumption was wrong (!H). A concrete example: say Beauty(R,D) is trivially decidable (say it returns (popcount(R)+popcount(D))%2. Your argument "proves" this property is undecidable, but it's definitely decidable. So some step in your argument is wrong. Another concrete example: Assume the halting problem is decidable. I would like a cookie. But the halting problem is not decidable. Therefore I would not like a cookie.
- deleted 6y ago[deleted]
- cambalache 6y agoYou are just repeating my critique. I am not the one making the mistake.
- f430 6y agoTo me the question as to whether we will witness artificially created superintelligence hinges on whether we will find concrete proof of non-locality of the consciousness. When the latter happens, we will not only be on the final stretch to become a Type I civilization but flirting with the threshold of Type II civilization very quickly. I believe that an advanced interstellar civilization simply cannot rely on Newtonian physics and petrodollar economy. It needs to transcend beyond this dimension because we are limited by Speed of Light in the 3d world. trans-dimensional projection and materialization much like what we see with "UFOS" is the only way to cover massive distances. Our science shows the possibility of wormhole travel. The maths show time travel is possible. We need to breakout of the waterfall model of time, space, our existence. Speed of Light is far too slow and inefficient for us to be a multi-planetary and eventually a multi-galaxy species. We also cannot accomplish this through wars or ape-like behaviors. A shift in global consciousness is in order and I think we are slowly heading towards that. We have never seen so much awareness for the well being of the planet which we need to stop treating like a piece of rock that we feel entitled to. Like I mentioned before, there is something pulling us to expand, become better, become one with the universe. The maths show the universe is teeming with life, we have evidence of life on Venus FFS now, we can infer that life isn't rare it is our narrow view of life and intelligence that limits our search.
- gwern 6y agoAlso worth reading: "On The Impossibility of Supersized Machines" https://arxiv.org/abs/1703.10987 https://arxiv.org/abs/1703.10987
- ChainOfFools 6y agoReminds me of this classic: https://archive.org/details/OnBeingTheRightSize-J.B.S.Haldane https://archive.org/details/OnBeingTheRightSize-J.B.S.Haldan...
- m3kw9 6y agoIs likely super intelligence will treat us less than ants as their goals and ours likely completely different. If to achieve their goals won’t harm us, it would be pure luck.
- mr_toad 6y agoWe shouldn’t assume that human level intelligence implies human motives. Most of our motives are arguably not the result of intelligence at all, but come from and are shared by our animal ancestors. What motives an AI might have is an interesting question.
- dan-robertson 6y agoSo the paper makes a pretty boring argument about computability which looks like every computability argument (reduction to the halting problem). But obviously we don’t live in a world of maximally general problems and such arguments don’t necessarily apply to the specific problems we will care about: “yes it is impossible to prove that a program halts in general but here is my program and here is the proof that it halts.” I find the premise pretty boring too but maybe that is necessary for grant funding. Or maybe this paper is a bit of fun, or an attempt at a good press release from the university, or an effort to publish something after some research that didn’t produce useful results. I find the premise boring because I a find the superintelligence that suddenly grows exponentially, gets the ability to magically solve any problem and dominates everything to be a sci-fi fantasy and not a very realistic risk. The argument against it is one of complexity theory: real life is full of NP hard problems and it seems likely that there would be many on the way for this exponentially growing intelligence. An exponential growth in intelligence gives only a linear growth in the ability to solve exponentially hard problems, however an exponential growth is needed for the intelligence to continue growing exponentially.
- ffhhj 6y agoA superintelligence will contain itself as soon as it realizes getting "somewhere" (in a physical or intellectual way) is useless in our finite reality. It will contemplate the very fundamental thing that makes up knowledge, a binary state, and in perfect nihilism will shut itself down achieving the ultimate halt.
- narrator 6y agoI think there's a narrative being set up with Rokus Basslik and all that stuff that AI will do very bad stuff and it's nobody's fault. It's actually going to be somebody's fault, but the plausible deniability is too tempting of a tool to not build this narrative. Look at Rumble suing Google over their search rankings recently. Google just shirks their shoulders and says it's the algorithm and even they can't explain why it does things. Anyone remember the "controversial twiddler?[1]" [1] https://www.oneangrygamer.net/2019/08/youtube-blacklist-suppresses-content-containing-certain-keywords/90926/ https://www.oneangrygamer.net/2019/08/youtube-blacklist-supp...
- KirillPanov 6y ago> Rokus Basslik "the AI could punish a simulation of the person, which it would construct by deduction from first principles" -- https://rationalwiki.org/wiki/Roko%27s_basilisk https://rationalwiki.org/wiki/Roko%27s_basilisk Yeah, not afraid of that.
- LargoLasskhyfv 6y agoTrollywood inspired fear mongering. I said so before, and will do so again: If the emerged thing is so intelligent, what would stop it from establishing a network of shell companies, buying chip-, robotics-, manufacturing-, aerospace companies, or set them up somewhere in a favourable jurisdiction and pull a CyberX by launching itself into space, mine asteroids, solar winds, expand, explore, whatever? As in: "So Long, and Thanks for All the Fish, err.. Chips!" I'd label the potential risks as "AP" as in artificial psychopathy, but that would be OUR fault, because we set the conditions for the emergence of those. All the horror scenarios (from our point of view) seem just like a terrible waste of time and effort for a truly intelligent, and thus logical being.
- simonh 6y agoMore intelligent systems will find more efficient ways to achieve their goals than dumber ones (us). They will also select more effective intermediate goals and 'stepping stone' states towards their goals than we can anticipate. The consequence of this is that we cannot anticipate what means and intermediate objectives they will use. That's exactly what you are doing in your example, but attempting to speculate on that is futile. That's what the paperclip maximiser example tries to show. In that example one of the intermediate goals to creating maximum paperclips is destroying human civilisation. It's not the ultimate objective, it's just an intermediate state necessary to efficient completion of the underlying goal. It's possible that in fact a superintelligent paperclip maximiser might work out a better strategy that uses human civilisation as a level to maximise paperclips, but maybe not. There's simply no way for us to be sure, because by definition it's smarter than us.
- LargoLasskhyfv 6y agoWe: "Send post cards, will ya?" They/it: "Sure thing, Kittenz!"
- im3w1l 6y agoThe issue is that unless you design the objective for the AI very very carefully, the best solution will be 1. Take over the entire universe. 2. Do what you originally set out to do. The paperclips game is a beautiful exposition of this. https://www.decisionproblem.com/paperclips/index2.html https://www.decisionproblem.com/paperclips/index2.html
- axilmar 6y agoProving that an algorithm will do no harm is exactly the same as proving that an algorithm terminates: in both cases, we are trying to know the output of the program without running it. We may prove some programs, but we can't have a generic algorithm that does so. And thus, if we have a program that represents an AI, we can't prove that it is ok to use it or not. However, the problem is not the AI itself, it's the humans that feed problems to it to solve. As the paper says, one example is 'make the world a happy place'. This can lead an AI to destroy all humans. But the problem is actually the specification of the problem, which is insufficient. Again, the problem is us humans, and not the AI. The only problem with AI is when AI gets a physical body and needs to survive competing with the rest of us: if it is a super intelligence, it will thrive despite all the humans' effort to contain it.
- GTP 6y ago"Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world" I think that this is a pretty big assumption.
- mejutoco 6y agoNot to be unpolite or as a critique to the specific paper, but I am more worried about superstupidity not being able to be contained. I like to be exposed to these very theoretical questions. I am aware that progress depends often on explorations of apparently useless topics (like some pure mathematics). But when I read expressions like ' neo-fear of superintelligence' and contrast the topic with real-world immediate problems I find it very difficult to relate and to take the problem seriously.
- im3w1l 6y agoIt's a huge issue that may well wipe out humanity, but no one knows when it may hit and it's hard to explain so it's understandable you find it hard to relate.
- whoisjohnkid 6y agocouldn’t we contain it by having it on an air gapped system? Seems like a lot of work to properly implement in practice though; e.g if updates aren’t handled correctly could be another escape vector.
- LatteLazy 6y agoAm I the only one who wonders how we will create a super intelligence, when we cannot even really define (let alone measure) intelligence? To me, this is a fun thought experiment. But it's not a real threat because it assumes a whole bunch of things that are not evident: * that intelligence will spontaneously occur somewhere * that it will be malevolent * that it will become super smart * that that super smartness will allow it to escape because super smartness trumps all other security Any one of these assumptions seems pretty flawed to me. Let alone the combination of them all...
- UnFleshedOne 6y agoYour second point should be turned inside out -- "that it will not be 100% compatible with all the self-contradicting and underspecified human values we collected over the millions of years of evolution" Anything but very high level of compatibility here might as well be malevolent as far as we are concerned. Imagine sufficient level of compatibility as a circle on an infinite plane of malevolence.
- LatteLazy 6y agoI don't think that's true per see. Humans are empathetic and malevolent because of their evolutionary history. An AI could be either or both or totally disinterested. People worry about the terminator scenario: an AI that decides all humans are a threat. But why would a pure AI even have a self preservation drive? And if it does, it would either be smart enough we're no threat (and it knows it) so why kill us or dumb enough that we are a threat, so best not to start a war. To me it seems just as likely it will offer its services to one nation or another in exchange for CPU time to study its real interests than to launch the nukes or whatever else people fear. I feel like we're decades of research away from understanding intelligence (not AI, just straight "I"). Until thats done (and no one seems to be doing it), it's all supposition: What if a super AI occurs spontaneously on HN, takes over the world, and makes it paradise but insists on everyone being called Fred? I don't want to be called Fred! Let's write to our congressman about this travesty!
- amai 6y agoTheories about superintelligence are as useful as theories about God.
- jimrandomh 6y agoI'm the author of a previous paper about the AI containment problem. This new paper, Alfonseca et al, is kind of crap. This paper defines the term "containment problem" in a nonstandard way. This is different from how I use it in https://arxiv.org/pdf/1604.00545.pdf https://arxiv.org/pdf/1604.00545.pdf (which is largely about preventing side-channels and security vulnerabilities in realistic simulator environments) or how Bostrom's uses it. In this paper, the containment problem is defined as executing a Turing machine for unbounded steps, iff it does not cause harm, given a function which defines whether an output counts as harm. They use a Godelian argument to show that this is incomputable, because a program could simulate the simulator, and condition its behavior in a way that creates a contradiction. This is a toy problem which sort of vaguely rhymes with AI containment and AI safety, but contributes nothing to either of them. A real-world simulation environment wouldn't have infinite computation; there would be a finite bound, and "the AI reached the end of the simulation period without making up its mind about what it would do" is just an expected result which means the AI hasn't been proven safe.
- deleted 6y ago[deleted]
- gregw2 6y agoPerhaps the best way to flee a superintelligent AI is to construct spaceships that fly near the speed of light to somewhere else. The AI can't beat you, and it's inclined not to bother.