3 ms·

In the "scientists complacent about dangers of the systems they are working on" file, I think we are all familiar with https://en.wikipedia.org/wiki/Demon_core

In the "scientists complacent about dangers of the systems they are working on" file, I think we are all familiar with https://en.wikipedia.org/wiki/Demon_core https://en.wikipedia.org/wiki/Demon_core. Now, admittedly, this was scientists being complacent about a thing they knew was dangerous, whereas here we are saying scientists don't think their thing is dangerous. But very clearly, AI could be dangerous, so it's more that these scientists don't think their system could be dangerous. Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. So maybe the problem is precisely that when we bring up extinction events from AGI, scientists rolling their eyes is the very reason we should be fucking worried. Their contempt for the possibility of the threat is what will get us killed.

ke88y 3y ago
> Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. Therac-25 is an excellent example, but of EXACTLY the opposite point. On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me. In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc. We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.
- lowbloodsugar 3y ago
  >while sci-fi about killer robots was entirely unhelpful in preventing software bugs. Unless we are now talking about actually building AI and killer robots, which we are assuredly doing.

ke88y 3y ago

> Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people. Therac-25 is an excellent example, but of EXACTLY the opposite point. On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me. In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc. We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.

lowbloodsugar 3y ago
>while sci-fi about killer robots was entirely unhelpful in preventing software bugs. Unless we are now talking about actually building AI and killer robots, which we are assuredly doing.