4 ms·

This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi sc

This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.

wongarsu 3y ago
At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong 1: https://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging
- Jerrrry 3y ago
  Is AI Safety a Pascal's Mugging? [Robert Miles AI Safety] https://www.youtube.com/watch?v=JRuNA2eK7w0 https://www.youtube.com/watch?v=JRuNA2eK7w0
Tumblewood 3y ago
This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
- pixl97 3y ago
  Existential risk is one of those problems that nearly impossible to measure in most cases. In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked. But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement. Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
zucker42 3y ago
Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite. If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
myrmidon 3y ago
If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!). So the following requirements: 1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century 2) AI acquires power/means to effect human extinction 3) AI decides against coexistence with humans Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...

wongarsu 3y ago

At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong 1: https://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging

Jerrrry 3y ago
Is AI Safety a Pascal's Mugging? [Robert Miles AI Safety] https://www.youtube.com/watch?v=JRuNA2eK7w0 https://www.youtube.com/watch?v=JRuNA2eK7w0

Tumblewood 3y ago

This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...

pixl97 3y ago
Existential risk is one of those problems that nearly impossible to measure in most cases. In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked. But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement. Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.

zucker42 3y ago

Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite. If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.

myrmidon 3y ago

If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!). So the following requirements: 1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century 2) AI acquires power/means to effect human extinction 3) AI decides against coexistence with humans Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...