3 ms·
A nonzero probability should be enough because the consequences are infinitely bad.
by amelius 2y ago
A nonzero probability should be enough because the consequences are infinitely bad.
- Buttons840 2y agoEvery path has non-zero risks. Choose your risk. If we outlaw AI, because of "risk", then people might experiment with it in secret, which has its own unique risks. If we study AI in the open, that has unique risks. If we descend further into authoritarianism to punish AI, that has risks. And no matter what we do about AI, the risks of nuclear war or life ending asteroid impacts remain.
- jncfhnb 2y agoGive me your life savings or I will destroy the universe
- DisgracePlacard 2y agohttps://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging > In philosophy, Pascal's mugging is a thought experiment demonstrating a problem in expected utility maximization. A rational agent should choose actions whose outcomes, when weighted by their probability, have higher utility. But some very unlikely outcomes may have very great utilities, and these utilities can grow faster than the probability diminishes. Hence the agent should focus more on vastly improbable cases with implausibly high rewards; this leads first to counter-intuitive choices, and then to incoherence as the utility of every choice becomes unbounded. Curiously enough, this idea can be traced to one of the most prominent AI Safety advocates.
- amelius 2y agoYes, probabilities are not the right tool for the job.