4 ms·

The way I see it is that we don't need to imagine any sci-fi scenario to see how any device with a computer (so basically anything from a phone to killer robots

The way I see it is that we don't need to imagine any sci-fi scenario to see how any device with a computer (so basically anything from a phone to killer robots or power plants) could go rogue or severely malfunction. Stuxnet was developed by skilled humans and was pretty successful. Personally I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10. Depending on the use case, the development could be 100% digital--the AI could either work it out in its mind, or run/rent a VM. And if there's a need for physical hardware that the superintelligence cannot emulate because of undocumented bugs and issues, it could hire someone with a garage to put the hw there and plug it--it doesn't require a factory and 10s or 100s of people. If you don't think it's a plausible scenario because I'm missing some step(s), then let me know!

gary_0 3y ago
> I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10 How will this Stuxnet^10 be able to do orders of magnitude more damage, though? Shut down power plants? Launch ze nukes? The "shut down power plants" issue already exists and people have been aware since the Y2K scare, and now due to cyberwarfare concerns. Such problems are solved by not being dumb, and making sure there are enough air-gaps and humans-in-the-loop that an errant computer command can't wreak havoc (or fire ze missiles). I guess your Stuxnet^10 could do damage by immobilizing the entire Tesla fleet, but once again, that's a dumb design issue with networked cars, not a vulnerability you need a super-AI to exploit. And I personally don't (and would never) drive a remotely-updatable car myself for exactly this reason. These are all problems that are easily solved by not doing anything monumentally stupid engineering-wise. The lesson we need to take to heart isn't Terminator, it's Chernobyl.
- TomaszZielinski 3y ago
  According to the Wikipedia article [1], Stuxnet not only infected the Iranian infrastructure but apparently also a Russian nuclear plant. And so air gaps & co. didn't help against a skilled, determined but still human-level attacker. But then I read the Wikipedia article more closely, and, for instance, it turns out that Stuxnet components were signed with stolen certificates so that Windows recognized them as legitimate. Which means that at best someone had not only write Stuxnet, but also get those certs. And even if it's easy in comparison, there had to be at least a remote hack involved, if not a physical one. And then there was the risk of those certs being revoked. Etc., etc. So, even if I'm more pessimistic than you, the lesson for me here is that if you start peeling the onion layers, it's all not so simple to pull off. [1] https://en.wikipedia.org/wiki/Stuxnet https://en.wikipedia.org/wiki/Stuxnet

gary_0 3y ago

> I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10 How will this Stuxnet^10 be able to do orders of magnitude more damage, though? Shut down power plants? Launch ze nukes? The "shut down power plants" issue already exists and people have been aware since the Y2K scare, and now due to cyberwarfare concerns. Such problems are solved by not being dumb, and making sure there are enough air-gaps and humans-in-the-loop that an errant computer command can't wreak havoc (or fire ze missiles). I guess your Stuxnet^10 could do damage by immobilizing the entire Tesla fleet, but once again, that's a dumb design issue with networked cars, not a vulnerability you need a super-AI to exploit. And I personally don't (and would never) drive a remotely-updatable car myself for exactly this reason. These are all problems that are easily solved by not doing anything monumentally stupid engineering-wise. The lesson we need to take to heart isn't Terminator, it's Chernobyl.

TomaszZielinski 3y ago
According to the Wikipedia article [1], Stuxnet not only infected the Iranian infrastructure but apparently also a Russian nuclear plant. And so air gaps & co. didn't help against a skilled, determined but still human-level attacker. But then I read the Wikipedia article more closely, and, for instance, it turns out that Stuxnet components were signed with stolen certificates so that Windows recognized them as legitimate. Which means that at best someone had not only write Stuxnet, but also get those certs. And even if it's easy in comparison, there had to be at least a remote hack involved, if not a physical one. And then there was the risk of those certs being revoked. Etc., etc. So, even if I'm more pessimistic than you, the lesson for me here is that if you start peeling the onion layers, it's all not so simple to pull off. [1] https://en.wikipedia.org/wiki/Stuxnet https://en.wikipedia.org/wiki/Stuxnet