6 ms·

Not OP, but I do work in reducing harms of deployed AI systems at a big tech company When most EA/Rationalist folks discuss AI Safety. They’re talking about AG

Not OP, but I do work in reducing harms of deployed AI systems at a big tech company When most EA/Rationalist folks discuss AI Safety. They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible Now there’s nothing inherently wrong with funding research of that sort. The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today. This existential measuring contest is just so odd and unnecessary. I really don’t care if you think AI Safety is a legitimate problem. I personally don’t but it’s when they begin to downplay others actual catastrophes that many of us take issue.

concordDance 4y ago
> I’ve had prominent EA folks tell me that Climate Change is not an existential risk Have you seen anything to indicate otherwise? The IPCC reports don't contain even a whiff in that direction. To the best of my knowledge zero people who know anything about climate change think its existential.
- andybak 4y ago
  Is there an accepted unambiguous definition of "existential" here? Is it: 1. All life on the planet dies 2. All advanced life on the planet dies? 3. All humans die? 4. Advanced civilization is destroyed irretrievably? 5. Advanced civilization is destroyed for a long period of time? etc And I'm not sure the group "people who know anything about climate change" are neccesarily better equipped to answer this question - it's a complex systems question. I'm not actually sure who is equipped to answer it accurately.
  - a3_nm 4y ago
    It looks like there are research works trying to define precisely this notion of existential risk in the context of climate change, see e.g. https://link.springer.com/article/10.1007/s10584-022-03430-y https://link.springer.com/article/10.1007/s10584-022-03430-y
  - concordDance 4y ago
    It's always 2 or 3 in my experience.
  - danaris 4y ago
    I'm not sure if this would actually count as "existential", but I think that it's well worth being very, very concerned about a consequence on the order of "Millions to billions of people die; large percentages of the rest are displaced and/or have their lives made significantly shorter, harsher, and less certain as war, famine, disease, and natural disasters become vastly more common around the globe." And there's really very little question that that's where climate change is leading if we don't get it under control soon.
quickthrower2 4y ago
The biggest threat of AGI IMO is people wanting and desiring to make the leap from human to digital. It is less The Matrix and more like plastic surgery or some other lifestyle enhancement. At first they wont, like people wouldn’t jump in an unlicensed taxi because that is silly, right!
FeepingCreature 4y ago
The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm. With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.
- danaris 4y ago
  Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe. From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge. This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.
  - FeepingCreature 4y ago
    I agree with your description of the AGI/ASI position. My take on it is the other way around: every step of the argument seems plainly obvious.
    - yifanl 4y ago
in3d 4y ago
They are right. Climate change is absolutely not an existential risk. “In a high-emissions scenario where little is done to curb planet-heating gases, global mortality rates will be raised by 73 deaths per 100,000 people by the end of the century” https://www.theguardian.com/us-news/2020/aug/04/rising-global-temperatures-death-toll-infectious-diseases-study https://www.theguardian.com/us-news/2020/aug/04/rising-globa...
- TheCoelacanth 4y ago
  "By the end of the century" is doing a lot of work in that statement.
- naasking 4y ago
  > Climate change is absolutely not an existential risk. That's an assessment based on what we know, but there's a lot we don't. For instance, it's possible that climate change could lead to a large scale collapse of food chains that could pose an existential risk to humanity. It's also plausible that climate change will lead to resource shortages that result in wars between nuclear powers, and the nuclear calamity that follows kills us all.
edanm 4y ago
> The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today. I'm sure some people do this, and it's bad. But to be clear, when you say some researchers are actively ignoring how to reduce harms of AI today... I'm not sure what you mean by "actively" ignore, but don't most researchers just by default ignore this, except for the few who are active in this field? In other words, I think it's totally legit to focus on long-term AGI risk, and totally ignore current short-term AI risk, as a personal career-choice, just like it's ok to not even work on AI at all. Unless someone is actively trying to cause less resources to go into short-term AI risk, what's the problem? > They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible I mean, we exist as (one would hope) intelligent beings. Why wouldn't AGI be possible? Your frame makes it seem like the default is that it isn't possible, which is wrong IMO.
naasking 4y ago
> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible I really don't understand AGI skeptics. Unless you think reality has some non-physical character that can't be reproduced outside of biology, it obviously follows that a physical machine can reproduce what another physical machine (human body) is doing.

concordDance 4y ago

> I’ve had prominent EA folks tell me that Climate Change is not an existential risk Have you seen anything to indicate otherwise? The IPCC reports don't contain even a whiff in that direction. To the best of my knowledge zero people who know anything about climate change think its existential.

andybak 4y ago
Is there an accepted unambiguous definition of "existential" here? Is it: 1. All life on the planet dies 2. All advanced life on the planet dies? 3. All humans die? 4. Advanced civilization is destroyed irretrievably? 5. Advanced civilization is destroyed for a long period of time? etc And I'm not sure the group "people who know anything about climate change" are neccesarily better equipped to answer this question - it's a complex systems question. I'm not actually sure who is equipped to answer it accurately.
- a3_nm 4y ago
  It looks like there are research works trying to define precisely this notion of existential risk in the context of climate change, see e.g. https://link.springer.com/article/10.1007/s10584-022-03430-y https://link.springer.com/article/10.1007/s10584-022-03430-y
- concordDance 4y ago
  It's always 2 or 3 in my experience.
- danaris 4y ago
  I'm not sure if this would actually count as "existential", but I think that it's well worth being very, very concerned about a consequence on the order of "Millions to billions of people die; large percentages of the rest are displaced and/or have their lives made significantly shorter, harsher, and less certain as war, famine, disease, and natural disasters become vastly more common around the globe." And there's really very little question that that's where climate change is leading if we don't get it under control soon.

quickthrower2 4y ago

The biggest threat of AGI IMO is people wanting and desiring to make the leap from human to digital. It is less The Matrix and more like plastic surgery or some other lifestyle enhancement. At first they wont, like people wouldn’t jump in an unlicensed taxi because that is silly, right!

FeepingCreature 4y ago

The atom bomb was once a hypothetical construct. The Einstein-Szilard letter was talking about a danger that they had no way of proving was even possible. Proof is irrelevant outside of mathematics; the only relevant question is cost to expected benefit - or expected harm. With dangerous research, when a conclusive demonstration of possibility is achieved, it is usually too late to do much in the way of prevention. That's why we don't expect the FDA to "prove" that harm from a new medication "is even possible" before regulating it; rather, we expect the pharmaceutical company to provide evidence for its expected harmlessness before we even make the attempt. Think of the FDA what you like, this is certainly a better standard than putting the burden of proof on harmful outcomes.

danaris 4y ago
Part of the problem with this is that conclusive demonstration that AGI is possible is not even in the same ballpark as a conclusive demonstration that it would have consequences like the EA folks seem to believe. From everything I've read about their views, they seem to believe that either the moment an AGI is created or very, very soon thereafter, it will achieve the Singularity and "ascend" to something near godhood from our perspective, and it will perceive humanity as a threat to it and immediately seek our destruction—which, because of its near-godhood, it will be able to achieve far faster than we have the ability to respond to. Thus, what we have to do is either prevent AGI from ever coming to pass (because as soon as it exists, it's effectively too late), or make absolutely certain that by the time it does, we have strong measures in place to either fight back against it, or get humanity the hell out of Dodge. This is built on such a foundation of shaky assumptions that giving it credibility is ludicrous.
- FeepingCreature 4y ago
  I agree with your description of the AGI/ASI position. My take on it is the other way around: every step of the argument seems plainly obvious.
  - yifanl 4y ago

in3d 4y ago

They are right. Climate change is absolutely not an existential risk. “In a high-emissions scenario where little is done to curb planet-heating gases, global mortality rates will be raised by 73 deaths per 100,000 people by the end of the century” https://www.theguardian.com/us-news/2020/aug/04/rising-global-temperatures-death-toll-infectious-diseases-study https://www.theguardian.com/us-news/2020/aug/04/rising-globa...

TheCoelacanth 4y ago
"By the end of the century" is doing a lot of work in that statement.
naasking 4y ago
> Climate change is absolutely not an existential risk. That's an assessment based on what we know, but there's a lot we don't. For instance, it's possible that climate change could lead to a large scale collapse of food chains that could pose an existential risk to humanity. It's also plausible that climate change will lead to resource shortages that result in wars between nuclear powers, and the nuclear calamity that follows kills us all.

edanm 4y ago

> The problem comes is that many EA/Rationalist folks begin downplaying legitimate tangible risks on the horizon (I’ve had prominent EA folks tell me that Climate Change is not an existential risk) or that many of these AI Safety researchers actively ignore how to reduce harms of the AI we have deployed today. I'm sure some people do this, and it's bad. But to be clear, when you say some researchers are actively ignoring how to reduce harms of AI today... I'm not sure what you mean by "actively" ignore, but don't most researchers just by default ignore this, except for the few who are active in this field? In other words, I think it's totally legit to focus on long-term AGI risk, and totally ignore current short-term AI risk, as a personal career-choice, just like it's ok to not even work on AI at all. Unless someone is actively trying to cause less resources to go into short-term AI risk, what's the problem? > They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible I mean, we exist as (one would hope) intelligent beings. Why wouldn't AGI be possible? Your frame makes it seem like the default is that it isn't possible, which is wrong IMO.

naasking 4y ago

> They’re talking about AGI, a hypothetical construct that we have no way of proving is even possible I really don't understand AGI skeptics. Unless you think reality has some non-physical character that can't be reproduced outside of biology, it obviously follows that a physical machine can reproduce what another physical machine (human body) is doing.