5 ms·

> One side seems to have much better quality of evidence. Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working

by ke88y 3y ago

> One side seems to have much better quality of evidence. Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk. They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun. So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about. Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters. Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think. > "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!! So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested. In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling. It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all. But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here. > What would you consider evidence of a significant AI risk? This is a really good question. I would consider a few things: 1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me. 2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s). 3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable). 4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways). I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.

ALittleLight 3y ago
Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything. Similarly, you think that x-risk dismissal among the experts you agree with is not self-interested - but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!" The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk and "experts" denying that are simply burying their head in the sand. AI, I use the term expansively, is making huge and rapid progress. We are approaching "human-level" intelligence and there is no guarantee, nor even any indication, that "human-level" is an upper limit. Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk. There are two basic kinds of risk. First, an AI powered tyranny where a small number of humans dominate everything enabled by AI systems they control - every security camera watched by a tireless intelligence, drones piloted by perfectly loyal intelligences, etc. Second, that the AI systems are not effectively controlled and pursue goals or objectives that are incompatible with human existence. In either case the fact that the AI brings superhuman intelligence to bear means that humanity at large will be overmatched in terms of capabilities. It is completely possible that neither of these scenarios come to pass, but the fact that one of the scenarios might come to pass is what makes the situation an existential risk - humanity eternally subjected by an irreversible dictatorship, or simply destroyed. I'm confused about your first item that would convince you of AI risk, but I think the others should all be considered met by ChatGPT. Of course, ChatGPT isn't currently useful in developing new WMDs - but it is absolutely useful in helping to bring someone up to speed on new topics in many different domains. As capabilities advance, why wouldn't ChatGPT using GPT-5, or 6, or 10 be able to helpfully guide the creation of new and better WMDs? AI isn't currently causing the problems you are worried about but certainly seems on track to do so soon. "Risk" doesn't mean that we are currently in the process of being destroyed by AI, but it does mean that there is a non-negligible possibility that we will soon find ourselves in that process.
- ke88y 3y ago
  > Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything. Of course. I was merely explaining the reason why my response would focus on x-risk rather than better surveys, and why. > but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!" Okay. But I have more to gain than to lose by advocating for AI Safety, since that's what I work on. > The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk My comment was about X-risk and AGI. Find any of a myriad of comments here where I agree there are real risks that should be taken seriously. Those risks -- the reasonable ones -- are not existential and have nothing to do with AGI. > Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk... I am not an adherent to this religious dogma and have seen no empirical evidence that the powerful rationalisms people use to talk themselves into these positions has any basis in current or future reality. I want the LessWrong cult to have fewer followers in the halls of power precisely because I actually do give a damn about preventing the worst-case FEASIBLE outcomes, which have nothing to do with AGI bullshit. BTW: if we're all too busy worrying about AGI who's going prevent the pile of ridiculous should-be-illegal bullshit that's about to cause a bunch of real harm? No one. If you want conspiracies about intent, look at who's funding all this AGI x-risk bullshit. And you can be sure as hell of one thing: no one in big tech wants me giving Congress advice on what our AI and data privacy regulations should look like. They would MUCH prefer endless hearings on "risks" they might even know are bullshit nonsense. But why focus any effort on 100% risks of data privacy instrusions, copyright infringement, wanton anti-trust violations, and large-scale disinfo campaigns when there's a 1% risk of extinction, right? But anyways. I'm just some fool who has actually spent over a decade in the trenches trying to prevent specific bad outcomes. I'm sure the sci-fi essays and prognostications from famous CEOs are far more persuasive and entertaining and convicting than the ramblings of some nobody actually try to fix real problems with mostly boring solutions.

by ke88y 3y ago

ALittleLight 3y ago

Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything. Similarly, you think that x-risk dismissal among the experts you agree with is not self-interested - but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!" The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk and "experts" denying that are simply burying their head in the sand. AI, I use the term expansively, is making huge and rapid progress. We are approaching "human-level" intelligence and there is no guarantee, nor even any indication, that "human-level" is an upper limit. Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk. There are two basic kinds of risk. First, an AI powered tyranny where a small number of humans dominate everything enabled by AI systems they control - every security camera watched by a tireless intelligence, drones piloted by perfectly loyal intelligences, etc. Second, that the AI systems are not effectively controlled and pursue goals or objectives that are incompatible with human existence. In either case the fact that the AI brings superhuman intelligence to bear means that humanity at large will be overmatched in terms of capabilities. It is completely possible that neither of these scenarios come to pass, but the fact that one of the scenarios might come to pass is what makes the situation an existential risk - humanity eternally subjected by an irreversible dictatorship, or simply destroyed. I'm confused about your first item that would convince you of AI risk, but I think the others should all be considered met by ChatGPT. Of course, ChatGPT isn't currently useful in developing new WMDs - but it is absolutely useful in helping to bring someone up to speed on new topics in many different domains. As capabilities advance, why wouldn't ChatGPT using GPT-5, or 6, or 10 be able to helpfully guide the creation of new and better WMDs? AI isn't currently causing the problems you are worried about but certainly seems on track to do so soon. "Risk" doesn't mean that we are currently in the process of being destroyed by AI, but it does mean that there is a non-negligible possibility that we will soon find ourselves in that process.

ke88y 3y ago
> Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything. Of course. I was merely explaining the reason why my response would focus on x-risk rather than better surveys, and why. > but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!" Okay. But I have more to gain than to lose by advocating for AI Safety, since that's what I work on. > The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk My comment was about X-risk and AGI. Find any of a myriad of comments here where I agree there are real risks that should be taken seriously. Those risks -- the reasonable ones -- are not existential and have nothing to do with AGI. > Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk... I am not an adherent to this religious dogma and have seen no empirical evidence that the powerful rationalisms people use to talk themselves into these positions has any basis in current or future reality. I want the LessWrong cult to have fewer followers in the halls of power precisely because I actually do give a damn about preventing the worst-case FEASIBLE outcomes, which have nothing to do with AGI bullshit. BTW: if we're all too busy worrying about AGI who's going prevent the pile of ridiculous should-be-illegal bullshit that's about to cause a bunch of real harm? No one. If you want conspiracies about intent, look at who's funding all this AGI x-risk bullshit. And you can be sure as hell of one thing: no one in big tech wants me giving Congress advice on what our AI and data privacy regulations should look like. They would MUCH prefer endless hearings on "risks" they might even know are bullshit nonsense. But why focus any effort on 100% risks of data privacy instrusions, copyright infringement, wanton anti-trust violations, and large-scale disinfo campaigns when there's a 1% risk of extinction, right? But anyways. I'm just some fool who has actually spent over a decade in the trenches trying to prevent specific bad outcomes. I'm sure the sci-fi essays and prognostications from famous CEOs are far more persuasive and entertaining and convicting than the ramblings of some nobody actually try to fix real problems with mostly boring solutions.