4 ms·

The most obvious paths to severe catastrophe begin with "AI gets to the level of a reasonably competent security engineer in general, and gets good enough to fi

The most obvious paths to severe catastrophe begin with "AI gets to the level of a reasonably competent security engineer in general, and gets good enough to find a security exploit in OpenSSL or some similarly widely used library". Then the AI, or someone using it, takes over hundreds of millions of computers attached to the internet. Then it can run millions of instances of itself to brute-force look for exploits in all codebases it gets its hands on, and it seems likely that it'll find a decent number of them—and probably can take over more or less anything it wants to. At that point, it has various options. Probably the fastest way to kill millions of people would involve taking over all internet-attached self-driving-capable cars (of which I think there are millions). A simple approach would be to have them all plot a course to a random destination, wait a bit for them to get onto main roads and highways, then have them all accelerate to maximum speed until they crash. (More advanced methods might involve crashing into power plants and other targets.) If a sizeable percentage of the crashes also start fires—fire departments are not designed to handle hundreds of separate fires in a city simultaneously, especially if the AI is doing other cyber-sabotage at the same time. Perhaps the majority of cities would burn. The above scenario wouldn't be human extinction, but it is bad enough for most purposes.

tech_ken 3y ago
How does "get okay at software engineering" entail that it is able to strategize at the level your scenario requires? Finding an OpenSSL exploit already seems like a big leap, but one that okay maybe I can concede is plausible. But then on top of that engineering and executing a series of events leading to the extinction of humanity? That's like an entirely different skillset, requiring plasticity, creativity, foresight, etc. Do we have any evidence that a big neural network is capable of this kind of behavior (and moreover capable of giving itself this behavior)? Especially when it's built for single-purpose uses (like an LLM)?
revelio 3y ago
That's not an obvious path at all: - Such exploits happen already and don't lead to extinction or really much more than annoyance for IT staff. - Most of the computers attached to the internet can't run even basic LLMs, let alone hypothetical super-intelligent AIs. - Very few cars (none?) let remote hackers kill people by controlling their acceleration. The available interfaces don't allow for that. Most people aren't driving at any given moment anyway. These scenarios all seem absurd.
- waterhouse 3y ago
  Addressing your points in order: - Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits. - You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible. - If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do. There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs https://www.youtube.com/watch?v=MK0SrxBC1xs

tech_ken 3y ago

How does "get okay at software engineering" entail that it is able to strategize at the level your scenario requires? Finding an OpenSSL exploit already seems like a big leap, but one that okay maybe I can concede is plausible. But then on top of that engineering and executing a series of events leading to the extinction of humanity? That's like an entirely different skillset, requiring plasticity, creativity, foresight, etc. Do we have any evidence that a big neural network is capable of this kind of behavior (and moreover capable of giving itself this behavior)? Especially when it's built for single-purpose uses (like an LLM)?

revelio 3y ago

That's not an obvious path at all: - Such exploits happen already and don't lead to extinction or really much more than annoyance for IT staff. - Most of the computers attached to the internet can't run even basic LLMs, let alone hypothetical super-intelligent AIs. - Very few cars (none?) let remote hackers kill people by controlling their acceleration. The available interfaces don't allow for that. Most people aren't driving at any given moment anyway. These scenarios all seem absurd.

waterhouse 3y ago
Addressing your points in order: - Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits. - You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible. - If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do. There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs https://www.youtube.com/watch?v=MK0SrxBC1xs