5 ms·

A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are alm

A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are almost certain to occur, but also for the most part have relatively minor consequences (mostly making things that are already happening worse). "Advanced AI will make humans extinct" is much less likely but has catastrophic consequences. Focusing on the catastrophic risks isn't unreasonable, especially since society at large seem to already handle the more frequently occurring risks (the EU's AI Act addresses many of them). And of course research into one of them benefits the other, so the categories aren't mutually exclusive.

pixl97 3y ago
I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues. As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing. Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.
silverlake 3y ago
This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.
- wongarsu 3y ago
  At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong 1: https://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging
  - Jerrrry 3y ago
    Is AI Safety a Pascal's Mugging? [Robert Miles AI Safety] https://www.youtube.com/watch?v=JRuNA2eK7w0 https://www.youtube.com/watch?v=JRuNA2eK7w0
- Tumblewood 3y ago
  This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
  - pixl97 3y ago
    Existential risk is one of those problems that nearly impossible to measure in most cases. In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked. But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement. Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
- zucker42 3y ago
  Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite. If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
- myrmidon 3y ago
  If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!). So the following requirements: 1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century 2) AI acquires power/means to effect human extinction 3) AI decides against coexistence with humans Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...
shafyy 3y ago
This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow. We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important. Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions. Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.
- HDThoreaun 3y ago
  Today's real issues are not that important compared to human extinction.
  - cocacola1 3y ago
    I’m not sure if extinction is a problem either. No one’s left to care about the issues, then.
    - hxugufjfjf 3y ago
      How long would the currently 17 people in space survive if every human on planet earth was eradicated?
      - cocacola1 3y ago
        Dunno. I imagine they’d follow soon, though.
  - jhanschoo 3y ago
    It seems to me that the most viable routes to human extinction are through superscaled versions of contemporary disasters: war, disease, and famine.
hackermatic 3y ago
Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.

pixl97 3y ago

I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues. As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing. Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.

silverlake 3y ago

This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.

wongarsu 3y ago
At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong 1: https://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging
- Jerrrry 3y ago
  Is AI Safety a Pascal's Mugging? [Robert Miles AI Safety] https://www.youtube.com/watch?v=JRuNA2eK7w0 https://www.youtube.com/watch?v=JRuNA2eK7w0
Tumblewood 3y ago
This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Existential_risk https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
- pixl97 3y ago
  Existential risk is one of those problems that nearly impossible to measure in most cases. In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked. But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement. Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
zucker42 3y ago
Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite. If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
myrmidon 3y ago
If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!). So the following requirements: 1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century 2) AI acquires power/means to effect human extinction 3) AI decides against coexistence with humans Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...

shafyy 3y ago

This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow. We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important. Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions. Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.

HDThoreaun 3y ago
Today's real issues are not that important compared to human extinction.
- cocacola1 3y ago
  I’m not sure if extinction is a problem either. No one’s left to care about the issues, then.
  - hxugufjfjf 3y ago
    How long would the currently 17 people in space survive if every human on planet earth was eradicated?
    - cocacola1 3y ago
      Dunno. I imagine they’d follow soon, though.
- jhanschoo 3y ago
  It seems to me that the most viable routes to human extinction are through superscaled versions of contemporary disasters: war, disease, and famine.

hackermatic 3y ago

Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.