5 ms·

On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts

On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts in the field. One side seems to have much better quality of evidence. "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! What would you consider evidence of a significant AI risk? From my point of view (x-risk believer) the arguments in favor of existential risk are obvious and compelling. That many experts in the field agree seems like validation of my personal judgement of the arguments. Surveys of researchers likewise seem to confirm this. What evidence do you think is lacking from this side that would convince you?

skepticATX 3y ago
How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals? Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?
- ke88y 3y ago
  >Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this? Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered. There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :) But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible. It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.
- ALittleLight 3y ago
  If we knew how specifically this would be built, then it would already be built. That's really the only way we could know if something would work - if it has, in fact, worked. As is, there are only ideas and possibilities. The trajectory that AI is on signals risk. AI capabilities are increasing, approaching human level, and nothing suggests AI capabilities will stop there. If you agree that a super intelligence - i.e. something smarter than humanity, could destroy or dominate humanity (either by itself or in collusion with a small number of humans) and you agree that we don't currently have a plan to prevent this from happening, then it seems to me that you also agree with AI x-risk.
ke88y 3y ago
> One side seems to have much better quality of evidence. Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk. They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun. So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about. Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters. Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think. > "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!! So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested. In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling. It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all. But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here. > What would you consider evidence of a significant AI risk? This is a really good question. I would consider a few things: 1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me. 2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s). 3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable). 4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways). I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.

skepticATX 3y ago

How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals? Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?

ke88y 3y ago
>Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this? Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered. There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :) But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible. It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.
ALittleLight 3y ago
If we knew how specifically this would be built, then it would already be built. That's really the only way we could know if something would work - if it has, in fact, worked. As is, there are only ideas and possibilities. The trajectory that AI is on signals risk. AI capabilities are increasing, approaching human level, and nothing suggests AI capabilities will stop there. If you agree that a super intelligence - i.e. something smarter than humanity, could destroy or dominate humanity (either by itself or in collusion with a small number of humans) and you agree that we don't currently have a plan to prevent this from happening, then it seems to me that you also agree with AI x-risk.

ke88y 3y ago

> One side seems to have much better quality of evidence. Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk. They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun. So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about. Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters. Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think. > "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!! So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested. In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling. It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all. But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here. > What would you consider evidence of a significant AI risk? This is a really good question. I would consider a few things: 1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me. 2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s). 3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable). 4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways). I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.