5 ms·

It feels like you are deliberately taking the most facile argument and ignoring the most substantive? As I outlined in the talk: for the breakthroughs that we

It feels like you are deliberately taking the most facile argument and ignoring the most substantive? As I outlined in the talk: for the breakthroughs that were required to do something that is thought of as pedestrian (namely, bringing up a computer), intelligence is simply not enough -- a superintelligence would not have suggested the breakthroughs because they required intense iteration, not simply the absorption of book knowledge. And yes, it's a technical argument, not an "intellectual argument" -- and as such requires much more technical depth to appreciate -- but that is very much the point, frankly...

gary_0 3y ago
> they required intense iteration, not simply the absorption of book knowledge I watched your talk, and I think this can be elaborated on even more to really drive it home. Because the big question is: how is your AI going to have agency, the ability to affect the world? How do we get from text on a screen, to Destroying All Humans? AI doomers seem to have this Ancient-Greek-philosophy view of science, that you can make progress just by sitting atop a marble column and pondering, but that's not how it works. You need to be able to poke and prod physical things, to experiment, and even if an AI was smart enough (and the high-level intellect required to do novel engineering is a pretty big handwave already) it has no means of poking and prodding. (You point this out in your talk by saying "here's all the hands-on trouble we had to go to just to boot a computer", but I think some people are missing the broader implications.) Given that the AI won't have an existing written corpus of "how to build killer robots" to draw on, or a physical corpus to use to develop novel technology (and troubleshoot it) it's pretty much up to the humans to develop the killer robots or whatever (and construct the killer robot assembly line), and then the blame is really on the humans for making autonomous murder machines and hooking them up to a psychopathic AI. (And even then, this is not so different from building a conventional army and then putting a psychopathic human in charge.) As you mention in your talk, a lot of other scenarios are just people wanting life to imitate Hollywood. I mean, you could hook up an AI to the nuclear Big Red Button, but that would just be dumb, and you wouldn't need an AI, just a plastic bird to press the "N" key whenever the system asked "start nuclear war?"
- bcantrill 3y ago
  Thank you for this. It was exactly my point in the talk, of course -- but in reading the comments here, I was very much beginning to doubt my own sanity...
  - TomaszZielinski 3y ago
    You could call me "AI serious complicationist" ("AI doomist" would probably be too much), and I still am as I'm writing this comment. But your point about the required amount of iteration and all the complications along the way are convincing for me-- I've realized that there might be more steps along the way to "serious complications" than I thought. So thank you for that!
  - Vecr 3y ago
    Have you tried to do some sort of sensitivity analysis on your world model? Something like the α-Maxmin rule for a Monte Carlo approximation?
- TomaszZielinski 3y ago
  The way I see it is that we don't need to imagine any sci-fi scenario to see how any device with a computer (so basically anything from a phone to killer robots or power plants) could go rogue or severely malfunction. Stuxnet was developed by skilled humans and was pretty successful. Personally I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10. Depending on the use case, the development could be 100% digital--the AI could either work it out in its mind, or run/rent a VM. And if there's a need for physical hardware that the superintelligence cannot emulate because of undocumented bugs and issues, it could hire someone with a garage to put the hw there and plug it--it doesn't require a factory and 10s or 100s of people. If you don't think it's a plausible scenario because I'm missing some step(s), then let me know!
  - gary_0 3y ago
    > I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10 How will this Stuxnet^10 be able to do orders of magnitude more damage, though? Shut down power plants? Launch ze nukes? The "shut down power plants" issue already exists and people have been aware since the Y2K scare, and now due to cyberwarfare concerns. Such problems are solved by not being dumb, and making sure there are enough air-gaps and humans-in-the-loop that an errant computer command can't wreak havoc (or fire ze missiles). I guess your Stuxnet^10 could do damage by immobilizing the entire Tesla fleet, but once again, that's a dumb design issue with networked cars, not a vulnerability you need a super-AI to exploit. And I personally don't (and would never) drive a remotely-updatable car myself for exactly this reason. These are all problems that are easily solved by not doing anything monumentally stupid engineering-wise. The lesson we need to take to heart isn't Terminator, it's Chernobyl.
TomaszZielinski 3y ago
You might have just planted an idea for a future super AI who to cont(r)act when iteration and manufacturing of that innocent Ring [1] is due :). I cannot decide how much I'm joking here (not about planting the idea, which obviously is a joke, but about the outsourcing part), it's an intriguing topic. [1] Totally not to rule them all

gary_0 3y ago

> they required intense iteration, not simply the absorption of book knowledge I watched your talk, and I think this can be elaborated on even more to really drive it home. Because the big question is: how is your AI going to have agency, the ability to affect the world? How do we get from text on a screen, to Destroying All Humans? AI doomers seem to have this Ancient-Greek-philosophy view of science, that you can make progress just by sitting atop a marble column and pondering, but that's not how it works. You need to be able to poke and prod physical things, to experiment, and even if an AI was smart enough (and the high-level intellect required to do novel engineering is a pretty big handwave already) it has no means of poking and prodding. (You point this out in your talk by saying "here's all the hands-on trouble we had to go to just to boot a computer", but I think some people are missing the broader implications.) Given that the AI won't have an existing written corpus of "how to build killer robots" to draw on, or a physical corpus to use to develop novel technology (and troubleshoot it) it's pretty much up to the humans to develop the killer robots or whatever (and construct the killer robot assembly line), and then the blame is really on the humans for making autonomous murder machines and hooking them up to a psychopathic AI. (And even then, this is not so different from building a conventional army and then putting a psychopathic human in charge.) As you mention in your talk, a lot of other scenarios are just people wanting life to imitate Hollywood. I mean, you could hook up an AI to the nuclear Big Red Button, but that would just be dumb, and you wouldn't need an AI, just a plastic bird to press the "N" key whenever the system asked "start nuclear war?"

bcantrill 3y ago
Thank you for this. It was exactly my point in the talk, of course -- but in reading the comments here, I was very much beginning to doubt my own sanity...
- TomaszZielinski 3y ago
  You could call me "AI serious complicationist" ("AI doomist" would probably be too much), and I still am as I'm writing this comment. But your point about the required amount of iteration and all the complications along the way are convincing for me-- I've realized that there might be more steps along the way to "serious complications" than I thought. So thank you for that!
- Vecr 3y ago
  Have you tried to do some sort of sensitivity analysis on your world model? Something like the α-Maxmin rule for a Monte Carlo approximation?
TomaszZielinski 3y ago
The way I see it is that we don't need to imagine any sci-fi scenario to see how any device with a computer (so basically anything from a phone to killer robots or power plants) could go rogue or severely malfunction. Stuxnet was developed by skilled humans and was pretty successful. Personally I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10. Depending on the use case, the development could be 100% digital--the AI could either work it out in its mind, or run/rent a VM. And if there's a need for physical hardware that the superintelligence cannot emulate because of undocumented bugs and issues, it could hire someone with a garage to put the hw there and plug it--it doesn't require a factory and 10s or 100s of people. If you don't think it's a plausible scenario because I'm missing some step(s), then let me know!
- gary_0 3y ago
  > I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10 How will this Stuxnet^10 be able to do orders of magnitude more damage, though? Shut down power plants? Launch ze nukes? The "shut down power plants" issue already exists and people have been aware since the Y2K scare, and now due to cyberwarfare concerns. Such problems are solved by not being dumb, and making sure there are enough air-gaps and humans-in-the-loop that an errant computer command can't wreak havoc (or fire ze missiles). I guess your Stuxnet^10 could do damage by immobilizing the entire Tesla fleet, but once again, that's a dumb design issue with networked cars, not a vulnerability you need a super-AI to exploit. And I personally don't (and would never) drive a remotely-updatable car myself for exactly this reason. These are all problems that are easily solved by not doing anything monumentally stupid engineering-wise. The lesson we need to take to heart isn't Terminator, it's Chernobyl.

TomaszZielinski 3y ago

You might have just planted an idea for a future super AI who to cont(r)act when iteration and manufacturing of that innocent Ring [1] is due :). I cannot decide how much I'm joking here (not about planting the idea, which obviously is a joke, but about the outsourcing part), it's an intriguing topic. [1] Totally not to rule them all