3 ms·

But the reason those incidents didn't become a lot worse was that the humans in the loop exercised sound judgment and common sense and had an ethical norm of no

by pdonis 3y ago

But the reason those incidents didn't become a lot worse was that the humans in the loop exercised sound judgment and common sense and had an ethical norm of not inadvertently causing a nuclear exchange. That's the GP's point: the risk is in what humans do, not what automated systems do. Even creating a situation where an automated system's wrong response is allowed to trigger a disastrous event because humans are taken out of the loop, is still a human decision; it won't happen unless humans who don't exercise sound judgment and common sense or who don't have proper ethical norms make such a disastrous decision. My biggest takeaway from all the recent events surrounding AI, and in fact from the AI hype in general, including hype about the singularity, AI existential risk, etc., is that I see nobody in these areas who qualifies under the criteria I stated above: exercising sound judgment and common sense and having proper ethical norms.

pixl97 3y ago
This is where things like drone swarms really put a kink in this whole ethical norms thing. I'm watching drones drop handgrenades from half the planet away in 4k on a daily basis. Moreso every military analysis out there says we need more of these and they need to control themselves so they can't be easily jammed. It's easy to say the future will be more of the same of what we have now, that is, if you ignore the people demanding an escalation of military capabilities.
- RandomLensman 3y ago
  Autonomous killing machines have been around for a long time and they remain highly effective - nothing really new there.
  - ben_w 3y ago
    True, but people are allowed to object to new ones. There's also a ban on at least some of the existing ones, after all: https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty
    - RandomLensman 3y ago
      An incomplete ban, though. Some people might object, others might want such new capabilities.
  - pixl97 3y ago
    Improvements in consumer AI Improve death AI. Saying nothing new here when you drop the weapon cost 10-100x is misunderstanding what leads to war.
    - RandomLensman 3y ago
      Drop in weapon costs are not really a major cause of war - or is there new research on that?
ben_w 3y ago
> sound judgment and common sense We only know their judgements were "sound" after the event. As for "common sense", that's the sound human brains make on the inside when they suffer a failure of imagination — it's not a real thing, it's just as much a hallucination as those we see in LLMs, and just as hard to get past when they happen: "I'm sorry, I see what you mean, $repeat_same_mistake". Which also applies to your next point: > Even creating a situation where an automated system's wrong response is allowed to trigger a disastrous event because humans are taken out of the loop, is still a human decision; it won't happen unless humans who don't exercise sound judgment and common sense or who don't have proper ethical norms make such a disastrous decision. Such humans are the norm. They are the people who didn't double-check Therac-25, the people who designed (and the people who approved the design of) Chernobyl, the people who were certain that attacking Pearl Harbour would take the USA out of the Pacific and the people who were certain that invading the Bay of Pigs would overthrow Castro, the people who underestimated Castle Bravo by a factor of 2.5 because they didn't properly account for Lithium-7, the people who filled the Apollo 1 crew cabin with pure oxygen and the people who let Challenger launch in temperatures below its design envelope. It's the Hindenburg, it's China's initial Covid response, it's the response to the Spanish Flu pandemic a century ago, it's Napoleon trying to invade Russia (and Hitler not learning any lesson from Napoleon's failure). It's the T-shirt company a decade ago who automated "Keep Calm and $dictionary_merge" until the wrong phrase popped out and the business had to shut down. It's the internet accidentally relying on npm left-pad, and it's every insufficiently tested line of code that gets exploited by a hacker. It's everyone who heard "Autopilot" and thought that meant they could sleep on the back seat while their Tesla did everything for them… and it's a whole heap of decisions by a whole bunch of people each of whom ought to have known better that ultimately led to the death of Elaine Herzberg. And, at risk of this list already being too long, it is found in every industrial health and safety rule as they are written in the blood of a dead or injured worker (or, as regards things like Beirut 2020, the public). Your takeaway shouldn't merely be that nobody "in the areas of AI or X-risk" has sound judgement, common sense, and proper ethical norms, but that no human does.
- RandomLensman 3y ago
  And, yet, humanity is still around with more people than ever on earth.

by pdonis 3y ago

pixl97 3y ago

This is where things like drone swarms really put a kink in this whole ethical norms thing. I'm watching drones drop handgrenades from half the planet away in 4k on a daily basis. Moreso every military analysis out there says we need more of these and they need to control themselves so they can't be easily jammed. It's easy to say the future will be more of the same of what we have now, that is, if you ignore the people demanding an escalation of military capabilities.

RandomLensman 3y ago
Autonomous killing machines have been around for a long time and they remain highly effective - nothing really new there.
- ben_w 3y ago
  True, but people are allowed to object to new ones. There's also a ban on at least some of the existing ones, after all: https://en.wikipedia.org/wiki/Ottawa_Treaty https://en.wikipedia.org/wiki/Ottawa_Treaty
  - RandomLensman 3y ago
    An incomplete ban, though. Some people might object, others might want such new capabilities.
- pixl97 3y ago
  Improvements in consumer AI Improve death AI. Saying nothing new here when you drop the weapon cost 10-100x is misunderstanding what leads to war.
  - RandomLensman 3y ago
    Drop in weapon costs are not really a major cause of war - or is there new research on that?

ben_w 3y ago

> sound judgment and common sense We only know their judgements were "sound" after the event. As for "common sense", that's the sound human brains make on the inside when they suffer a failure of imagination — it's not a real thing, it's just as much a hallucination as those we see in LLMs, and just as hard to get past when they happen: "I'm sorry, I see what you mean, $repeat_same_mistake". Which also applies to your next point: > Even creating a situation where an automated system's wrong response is allowed to trigger a disastrous event because humans are taken out of the loop, is still a human decision; it won't happen unless humans who don't exercise sound judgment and common sense or who don't have proper ethical norms make such a disastrous decision. Such humans are the norm. They are the people who didn't double-check Therac-25, the people who designed (and the people who approved the design of) Chernobyl, the people who were certain that attacking Pearl Harbour would take the USA out of the Pacific and the people who were certain that invading the Bay of Pigs would overthrow Castro, the people who underestimated Castle Bravo by a factor of 2.5 because they didn't properly account for Lithium-7, the people who filled the Apollo 1 crew cabin with pure oxygen and the people who let Challenger launch in temperatures below its design envelope. It's the Hindenburg, it's China's initial Covid response, it's the response to the Spanish Flu pandemic a century ago, it's Napoleon trying to invade Russia (and Hitler not learning any lesson from Napoleon's failure). It's the T-shirt company a decade ago who automated "Keep Calm and $dictionary_merge" until the wrong phrase popped out and the business had to shut down. It's the internet accidentally relying on npm left-pad, and it's every insufficiently tested line of code that gets exploited by a hacker. It's everyone who heard "Autopilot" and thought that meant they could sleep on the back seat while their Tesla did everything for them… and it's a whole heap of decisions by a whole bunch of people each of whom ought to have known better that ultimately led to the death of Elaine Herzberg. And, at risk of this list already being too long, it is found in every industrial health and safety rule as they are written in the blood of a dead or injured worker (or, as regards things like Beirut 2020, the public). Your takeaway shouldn't merely be that nobody "in the areas of AI or X-risk" has sound judgement, common sense, and proper ethical norms, but that no human does.

RandomLensman 3y ago
And, yet, humanity is still around with more people than ever on earth.