3 ms·
> sound judgment and common sense We only know their judgements were "sound" after the event. As for "common sense", that's the sound human brains make on the
by ben_w 3y ago
> sound judgment and common sense
We only know their judgements were "sound" after the event. As for "common sense", that's the sound human brains make on the inside when they suffer a failure of imagination — it's not a real thing, it's just as much a hallucination as those we see in LLMs, and just as hard to get past when they happen: "I'm sorry, I see what you mean, $repeat_same_mistake".
Which also applies to your next point:
> Even creating a situation where an automated system's wrong response is allowed to trigger a disastrous event because humans are taken out of the loop, is still a human decision; it won't happen unless humans who don't exercise sound judgment and common sense or who don't have proper ethical norms make such a disastrous decision.
Such humans are the norm. They are the people who didn't double-check Therac-25, the people who designed (and the people who approved the design of) Chernobyl, the people who were certain that attacking Pearl Harbour would take the USA out of the Pacific and the people who were certain that invading the Bay of Pigs would overthrow Castro, the people who underestimated Castle Bravo by a factor of 2.5 because they didn't properly account for Lithium-7, the people who filled the Apollo 1 crew cabin with pure oxygen and the people who let Challenger launch in temperatures below its design envelope. It's the Hindenburg, it's China's initial Covid response, it's the response to the Spanish Flu pandemic a century ago, it's Napoleon trying to invade Russia (and Hitler not learning any lesson from Napoleon's failure). It's the T-shirt company a decade ago who automated "Keep Calm and $dictionary_merge" until the wrong phrase popped out and the business had to shut down. It's the internet accidentally relying on npm left-pad, and it's every insufficiently tested line of code that gets exploited by a hacker. It's everyone who heard "Autopilot" and thought that meant they could sleep on the back seat while their Tesla did everything for them… and it's a whole heap of decisions by a whole bunch of people each of whom ought to have known better that ultimately led to the death of Elaine Herzberg. And, at risk of this list already being too long, it is found in every industrial health and safety rule as they are written in the blood of a dead or injured worker (or, as regards things like Beirut 2020, the public).
Your takeaway shouldn't merely be that nobody "in the areas of AI or X-risk" has sound judgement, common sense, and proper ethical norms, but that no human does.
- RandomLensman 3y agoAnd, yet, humanity is still around with more people than ever on earth.
- ben_w 3y agoIf they'd wiped us out, we wouldn't be here to argue about it. We can look at the small mistakes that only kill a few, and pass rules to prevent them; we can look at close calls for bigger disasters (there were a lot of near misses in the Cold War); we can look at how frequency scales with impact, and calculate an estimated instantaneous risk for X-risks; but one thing we can't do is forecast the risk of tech that has yet to be invented. We can't know how many (or even which specific) safety measures are needed to prevent extinction by paperclip maximiser unless we get to play god with a toy universe where the experiment can be run many times — which doesn't mean "it will definitely go wrong", it could equally well mean our wild guess about what safety looks like has one weird trick that will make all AI safe but we don't recognise that trick and then add 500 other completely useless requirements on top of it that do absolutely nothing. We don't know, we're not smart enough to know.
- RandomLensman 3y agoExactly. Wasting large efforts on de-risking purely hypothetical technology isn't what got us to where we are now.
- ben_w 3y agoThe people working on the Manhattan Project did more than zero de-risking about nukes while turning them from hypothetical to real.
- RandomLensman 3y agoAt that time there was nothing hypothetical about them anymore. They were known to be feasible and practical, not even requiring a test for the Uranium version.