3 ms·

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop a

We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. (The same playground reasoning the EA community engages in, which is a large vector for the xrisk hype. Just multiply by a large enough number, and you will surely have the biggest number) To the credit of the authors, they don't engage in that. There is no hand wringing over the absolutely unlikely case of "but what if the AI awakens". But it's still an extremely weak study - it proves nothing (none of the results are statistically significant), and even if it had shown significant uplift, it's meaningless without a control. Of course people who have access to a knowledge store do slightly better than people who don't. I'm willing to bet that "access to a 10-book research library" produces roughly the same uplift. Without that (trivial) control, it's really bad study design. And the moment you take this study and its non-results and call it "Building an early warning system for LLM-aided biological threat creation", you've absolutely lost all credibility.

foo3a9c4 3y ago
Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing. > We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused. > xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth? > All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development. Premise 1. I value my own survival over just about anything else. Premise 2. If an existential catastrophe occurs, then I will die. Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe. Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.
- groby_b 3y ago
  Premise 3 is where the problem is, of course. We have no idea how to build AGI. We know LLMs won't be it. Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI. Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence. Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from. This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered". Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is. LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop. Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.
  - foo3a9c4 3y ago
    Thank you for the thoughtful reply. > Premise 3 is where the problem is, of course. I don't believe premise 3 is a problem exactly, but I do believe that it is a non-trivial challenge to determine whether or not it is true. > We have no idea how to build AGI. We know LLMs won't be it. > Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence. > Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from. > This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered". It looks to me that you are making the following argument: Premise G1. Humans do not currently know how to build AGI. Premise G2. It might be impossible to build ASI. Premise G3. It is unclear how likely an ASI is to cause an existential catastrophe. Conclusion. There is not a significant chance of catastrophe from ASI. I believe that argument is about an important point (chance of AI catastrophe) and that it is a pretty good argument. But the original premise 3 says, "If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.", so AFAICT your argument doesn't substantively address it. (ie, your argument's conclusion doesn't tell me anything about whether or not premise 3 is true) I apologize if I have misunderstood your point. > Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI. We may be using the word "alignment" slightly differently. By "alignment" I just meant getting the algorithmic system to have precisely the goal that its human programmers want it to have. I would call, for example, RLHF a "tool" for trying to achieve alignment. How do you want to use the terms "alignment" and "alignment tool" going forward in the discussion? > Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is. I agree that there are other plausible sources of catastrophe for humans, to name a few others: asteroids, supervolcanoes and population collapse. I understand you to be making a new point now, but I just want to state that I do not believe the existence of other plausible existential threats to be a rebuttal of premise 3. > LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop. I agree that LLMs may plausibly cause significant harm in the short term via disinformation and unemployment. And again, I understand you to be making a new point, but I just want to state that I do not believe the plausibility of such LLM harms is a rebuttal against premise 3. > Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines. I'm not sure who you mean by "we" here, so I'm not sure if your claim about them is true or not.
hollerith 3y ago
>the cost is infinity, so any risk is a danger That's not the argument. The argument is that human extinction is what you would naturally expect to happen if AI research continues on its present course unless you are biased because your income depends on AI research continuing unimpeded or you have an irrational emotional need to believe that technological progress is always good or you considered the question for 3 minutes then held stubbornly to the conclusions of that 3 minutes of thinking. When sci-fi authors for example have treated the topic in fiction (e.g., Vinge, Greg Bear, James Cameron's Terminator) most of the time the AI wipes out the species that created it.
- groby_b 3y ago
  "what you would naturally expect to happen " Why? What is the reasoning this "is naturally expected" "When sci-fi authors for example have treated the topic in fiction" I'm sorry, but what you read in that book, saw in that movie isn't actually science. It's a cautionary tale about humans and what they are willing to do.
  - digging 3y ago
    > Why? What is the reasoning this "is naturally expected" There are many articles written on such a topic. In short, we have no way of predicting how an AGI will think, and there are more pathways to it being our enemy (intentionally or not) than to it being our ally. Especially since we can't even conceive of what it would look like for an entity to be the ally of all of humanity - humanity itself is not united on any goal at all. Pick one goal. Any goal that does or could affect humanity on a global scale. Now try to work out a plan to achieve that goal. Does your plan have the potential to anger a military power? If yes, you're a threat to humanity if you try to enact that goal. Even beyond the reasoning that AGI is likely to be dangerous, imagine it's a just 50/50 chance. Or even a 10% chance. Even a 5% chance. How low does the chance of human extinction need to go before you're willing to press The Button? Most of the arguments I see here amount to either "There is literally no risk of superhuman AI threatening human extinction," which is unequivocally wrong, or "There is literally no possibility of AGI existing," which is also unequivocally wrong. People usually say, "well it's at least decades away," which is actually them in denial that AGI can exist and be an existential threat. Because if they really believed it could happen in a few decades it would still be worth working on. Imagine someone told you "In 40 years a superhuman AGI will awaken and flip a coin to decide whether or not it destroys humanity," how long would you wait to start working on defense?

foo3a9c4 3y ago

Thanks for engaging in a discussion about AIXR. IMO it's important to figure out if we are actually about to kill ourselves or whether some people are just getting worked up over nothing. > We should also deeply worry about space aliens showing up and blasting us out of the sky. If they're sufficiently powerful, that could absolutely happen! Stop any radio emissions! If I believed that dangerous space aliens were likely, then I would be interested in investigating ways to avert/survive such an encounter. This seems pretty rational to me, but maybe I'm confused. > xRisk is an absolutely stupid way to reason about AI. It's an unprovable risk that requires "mitigation just in case". By "unprovable risk" do you mean that it's literally impossible to know anything about the likelihood that dangerous algorithms could kill (nearly) all people on Earth? > All this is is saying "but if it were to happen, the cost is infinity, so any risk is a danger! Infinity times anything is infinity!". It's playground reasoning. Maybe you've seen people make that argument, but it strikes me as a strawman. Here is what I consider to be a better argument for not rushing ahead with capabilities development. Premise 1. I value my own survival over just about anything else. Premise 2. If an existential catastrophe occurs, then I will die. Premise 3. If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe. Conclusion. So, I strongly prefer that ASI not be built until alignment is understood.

groby_b 3y ago
Premise 3 is where the problem is, of course. We have no idea how to build AGI. We know LLMs won't be it. Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI. Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence. Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from. This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered". Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is. LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop. Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines.
- foo3a9c4 3y ago
  Thank you for the thoughtful reply. > Premise 3 is where the problem is, of course. I don't believe premise 3 is a problem exactly, but I do believe that it is a non-trivial challenge to determine whether or not it is true. > We have no idea how to build AGI. We know LLMs won't be it. > Even if we create AGI, we have no indication it is possible to build a orders-of-magnitude more "intelligent" thing. This is predicated entirely on the notion that if you can do it at scale, you get more, and there's no evidence thinking more makes for more intelligence. > Even if that were possible and we build an ASI, it's not at all clear this would lead to existential catastrophe. An ASI is presumably smart enough to see it's about to end the world as we know it, and knows where its power supply comes from. > This leaves us with an xrisk probability so close to zero it's virtually indistinguishable from zero. The only way to make it mean anything is "let's multiply it with infinity" - "it will end humanity, and my own survival is endangered". It looks to me that you are making the following argument: Premise G1. Humans do not currently know how to build AGI. Premise G2. It might be impossible to build ASI. Premise G3. It is unclear how likely an ASI is to cause an existential catastrophe. Conclusion. There is not a significant chance of catastrophe from ASI. I believe that argument is about an important point (chance of AI catastrophe) and that it is a pretty good argument. But the original premise 3 says, "If ASI is built before alignment is understood, then there is a significant chance of existential catastrophe.", so AFAICT your argument doesn't substantively address it. (ie, your argument's conclusion doesn't tell me anything about whether or not premise 3 is true) I apologize if I have misunderstood your point. > Alignment is a tool that works with LLMs, but we don't know if it will work for whatever produces AGI. We may be using the word "alignment" slightly differently. By "alignment" I just meant getting the algorithmic system to have precisely the goal that its human programmers want it to have. I would call, for example, RLHF a "tool" for trying to achieve alignment. How do you want to use the terms "alignment" and "alignment tool" going forward in the discussion? > Meanwhile, ordinary humans can use currently existing tools to end the world just fine. Nukes are readily available. We're obviously not really interested in public health. Climate refugees will be a giant problem soon-ish. The economy is very much a house of cards, but a house of cards that keeps society functioning as-is. I agree that there are other plausible sources of catastrophe for humans, to name a few others: asteroids, supervolcanoes and population collapse. I understand you to be making a new point now, but I just want to state that I do not believe the existence of other plausible existential threats to be a rebuttal of premise 3. > LLMs are a fantastic disinfo tool right now. There's a reasonably good chance they will calcify biases. They will cause large economic damage because 1) they lift up the baseline of work, and 2) they're just good enough that there's economic incentive to replace workers with it, but 3) they're shitty enough that the resulting output will ultimately be worse because we removed humans from the loop. I agree that LLMs may plausibly cause significant harm in the short term via disinformation and unemployment. And again, I understand you to be making a new point, but I just want to state that I do not believe the plausibility of such LLM harms is a rebuttal against premise 3. > Those are actual risks. That we sweep under the carpet, because "xrisk" makes for much more grabby headlines. I'm not sure who you mean by "we" here, so I'm not sure if your claim about them is true or not.

hollerith 3y ago

>the cost is infinity, so any risk is a danger That's not the argument. The argument is that human extinction is what you would naturally expect to happen if AI research continues on its present course unless you are biased because your income depends on AI research continuing unimpeded or you have an irrational emotional need to believe that technological progress is always good or you considered the question for 3 minutes then held stubbornly to the conclusions of that 3 minutes of thinking. When sci-fi authors for example have treated the topic in fiction (e.g., Vinge, Greg Bear, James Cameron's Terminator) most of the time the AI wipes out the species that created it.

groby_b 3y ago
"what you would naturally expect to happen " Why? What is the reasoning this "is naturally expected" "When sci-fi authors for example have treated the topic in fiction" I'm sorry, but what you read in that book, saw in that movie isn't actually science. It's a cautionary tale about humans and what they are willing to do.
- digging 3y ago
  > Why? What is the reasoning this "is naturally expected" There are many articles written on such a topic. In short, we have no way of predicting how an AGI will think, and there are more pathways to it being our enemy (intentionally or not) than to it being our ally. Especially since we can't even conceive of what it would look like for an entity to be the ally of all of humanity - humanity itself is not united on any goal at all. Pick one goal. Any goal that does or could affect humanity on a global scale. Now try to work out a plan to achieve that goal. Does your plan have the potential to anger a military power? If yes, you're a threat to humanity if you try to enact that goal. Even beyond the reasoning that AGI is likely to be dangerous, imagine it's a just 50/50 chance. Or even a 10% chance. Even a 5% chance. How low does the chance of human extinction need to go before you're willing to press The Button? Most of the arguments I see here amount to either "There is literally no risk of superhuman AI threatening human extinction," which is unequivocally wrong, or "There is literally no possibility of AGI existing," which is also unequivocally wrong. People usually say, "well it's at least decades away," which is actually them in denial that AGI can exist and be an existential threat. Because if they really believed it could happen in a few decades it would still be worth working on. Imagine someone told you "In 40 years a superhuman AGI will awaken and flip a coin to decide whether or not it destroys humanity," how long would you wait to start working on defense?