6 ms·
The law of unintended consequences - in this case, you take a simplified, abstract model (operating system, processes, memory) of what is really happening, and
by mannykannot 10y ago
The law of unintended consequences - in this case, you take a simplified, abstract model (operating system, processes, memory) of what is really happening, and you modify it to produce a different abstract model (the above in containers), and it looks relatively simple and straightforward, but the abstractions blind you to what is really going on, which is far from that.
So good luck, humanity, in keeping AI under control, should it be developed.
- Filligree 10y agoIt's fine, it's fine. We'll just program them to never harm humans or, through inaction, allow humans to come to any sort of harm, no matter how minor.
- Animats 10y agoWhat caused this, perhaps, was continuous OS distribution updating. When OS distributions were updated once a year or so, and there was support for one version back, everybody developed to the same target. Once a year you had to update your code, and you had a year to get ready. Containers, and version pinning, are ways to obtain stability in the face of continuous updating.
- bmh100 10y ago> So good luck, humanity, in keeping AI under control, should it be developed. This seems to anthropomorphize AI. Why should we expect AI in general to be difficult to control? AlphaGo, while not a strong AI, seems quite harmless.
- mikeash 10y agoAlphaGo is still dumber than a potted plant, the tiny amount of intelligence it has is just all focused on Go, which is enough to beat the masters at that one specific task. Now imagine you have a system with human-level intelligence and you tell it to beat the masters. Rather than becoming really good at Go, it might discover how to escape its container and sabotage the match, murder its opponents, etc.
- bmh100 10y agoOkay, I will go along with your scenario. Suppose we have a system with human-level intelligence, according to your definition, and we set up the following conditions: 1. The AI is embodied in a computer the size of a laptop. 2. The AI is given the objective "increase your skill in Go as much as possible and play to the best of your ability when presented with a specific position". 3. The computer has one input: an electronic Go board with a button to submit the position to the computer. 4. The computer has one output: a second electronic Go board. 5. The electronic Go boards indicate the status of positions by rotating cubes to white, black, or beige ("no piece") sides with a slow servo motor for each position. Under what conditions would this AI escape its container or murder its opponents?
- mikeash 10y agoThe AI decides that the best way to become better at Go would be to obtain control of more computing power. It carries out an exploit on its own container which gives it root access on the local computer, then spreads on to the internet. Once it takes over all existing computers, the next logical step to become better at Go is to turn all available non-computer matter into more computers. Being made of matter, humans may find this objectionable. (I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)
- bmh100 10y agoYou have identified a chain of events. Here is a list of questions I have about how that chain of events occurs. I am asking these in good faith, not being snarky, just in case it seemed otherwise. How would the AI learn that more processing power equals becoming better at Go? How would the AI learn to program? How would the AI acquire the creativity necessary to develop novel programs? Why would a Go AI have the ability to modify its own code? How would it learn to use such an ability to escape its container? How would it identify escaping its container as a goal? How did this computer become connected to the internet? How would it learn to communicate over a network? How would it learn that taking over computers would give it more processing power? Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
- mannykannot 10y agoThe comment is somewhat tongue-in-cheek, but I think the concern is valid. It probably is anthopomophizing, but that does not mean that it is automatically wrong. It might be game over by the time an example is apparent.
- bmh100 10y agoFor an AI to become "uncontrolled" requires quite a lot of assumptions. I don't dismiss the concern as invalid, but I don't think the assumptions implied by the question are necessarily plausible either.
- mannykannot 10y agoThe argument follows from the first thing I wrote in this thread. Our understanding of complex systems is through abstractions, but abstractions (by definition) hide the details. Just as attackers can exploit details that are 'below the radar' of an abstract view, AI with a mind of its own (which it would have, by definition) could probably do things that escape our attention, including social engineering. That is not necessarily bad, but it might be out of our control. To be clear: I am considering real AI, not that which is sometimes passed off as AI these days.
- bmh100 10y agoHow would a strong AI learn to influence humans? How would it come to the conclusion that influencing humans would achieve its goal? How would it develop the ability to socially engineer humans in useful ways before the people monitoring it would notice? How would a strong AI develop the "motivation", for lack of a better term, to influence humans to remove constraints on its behavior? Why would this strong AI even decide that removing constraints would be its "goal" in the first place?