3 ms·
This has a Pascal's wager sort of feel. The possible negative consequences of a rampant AI are infinite, so we all ought to take it Very Seriously. The trouble
by dTal 3y ago
This has a Pascal's wager sort of feel. The possible negative consequences of a rampant AI are infinite, so we all ought to take it Very Seriously. The trouble is that there are lots of other scenarios with unbounded negative value - for instance, tight centralized control of technology supports a stable dystopia that keeps nearly all humans in a perpetual state of suffering. It's not at all clear, when you add it all up, that "prevent 'misaligned' AIs from being deployed at all costs" is the correct strategy to minimize risk.
- circuit10 3y agoThe difference to Pascal’s wager is that with that the probability is vanishingly small, but given current progress the probability of AI causing an existential threat is actually quite high. You can’t just call anything that has very bad negative consequences Pascal’s wager Video about this: “Is AI Safety a Pascal’s Mugging?” https://youtu.be/JRuNA2eK7w0 https://youtu.be/JRuNA2eK7w0
- circuit10 3y agoI sort of mixed up “wager” and “mugging” here but you get the point
- dTal 3y agoNot at all, "Pascal's mugging" was exactly the term I would have used, had I remembered it :) That video is a very good reply. One response that comes to mind is that there is an implicit subtext behind any discussion of AI safety that we must do something about it, and soon. A good "anti-god" for his payoff matrix would be "the act of calling for AI safety will inadvertently be used as political capital by those wishing to be the gatekeepers of powerful technology" - the chances of that are quite high, and the negative consequences quite bad, if not necessarily apocalyptic.