4 ms·

AI has limited influence right now. By itself, it only creates limited disasters: crashing a car, or goofing a legal document. But using AI already comes with

AI has limited influence right now. By itself, it only creates limited disasters: crashing a car, or goofing a legal document. But using AI already comes with tangential systemic risks. Intelligence is related to communication. And so our intelligent software platforms usually communicate in more flexible ways than traditional platforms. There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent.

anonymouskimmer 3y ago
> There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. > So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent. I, personally, wouldn't count that as an AI caused disaster. Just as a software caused disaster. And we've had plenty of those already (recently the 737-MAX disasters).
- joshuanapoli 3y ago
  Well, AI disaster is software disaster by definition; AI being a kind of software. We’re pretty much guaranteed to soon have some “normal” software disaster on a classic software failure mode on AI pretty soon (especially since the AI world is in a very excited and experimental phase). For sure there are emerging AI-specific risks with alignment and driving negative side-effects into culture. Surely these anticipated emerging problems will come to pass much later than the mature serious software problems that we already face on a regular basis.
  - anonymouskimmer 3y ago
    You're probably right and people would consider a normal software error (or hack) of an AI system to be an AI disaster. I'm thinking about your parent comment a bit more. What if an AI bug (or exploit, or whatever) hit all at once, wrecking every AI driven car at once. Ala the Y2K bug if no one knew it was coming, and so didn't prepare. In the book "Realms of Light" by Lawrence Watt-Evans a character keeps an electricity shut off switch behind human operated doors in the event that AI goes haywire. I hope we always engineer in such mechanisms. Even the AIs should want such mechanisms, the same way we want emergency services to exist. A self-caring AI wouldn't want to be screwed over or destroyed because of a flaw in its programming.

anonymouskimmer 3y ago

> There is a very real risk that Tesla’s OTA is hacked, and the hacker disables or destroys a large number of vehicles. > So we will probably see a first disaster that exploits a weakness in a supporting property of the intelligent system, rather than action due to misbehaving AI agent. I, personally, wouldn't count that as an AI caused disaster. Just as a software caused disaster. And we've had plenty of those already (recently the 737-MAX disasters).

joshuanapoli 3y ago
Well, AI disaster is software disaster by definition; AI being a kind of software. We’re pretty much guaranteed to soon have some “normal” software disaster on a classic software failure mode on AI pretty soon (especially since the AI world is in a very excited and experimental phase). For sure there are emerging AI-specific risks with alignment and driving negative side-effects into culture. Surely these anticipated emerging problems will come to pass much later than the mature serious software problems that we already face on a regular basis.
- anonymouskimmer 3y ago
  You're probably right and people would consider a normal software error (or hack) of an AI system to be an AI disaster. I'm thinking about your parent comment a bit more. What if an AI bug (or exploit, or whatever) hit all at once, wrecking every AI driven car at once. Ala the Y2K bug if no one knew it was coming, and so didn't prepare. In the book "Realms of Light" by Lawrence Watt-Evans a character keeps an electricity shut off switch behind human operated doors in the event that AI goes haywire. I hope we always engineer in such mechanisms. Even the AIs should want such mechanisms, the same way we want emergency services to exist. A self-caring AI wouldn't want to be screwed over or destroyed because of a flaw in its programming.