10 ms·

> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those

by ke88y 3y ago

> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate. Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make. Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder. I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement. But even within that sample: > The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. Ie even lower than my guess of 90% above.

arisAlexis 3y ago
Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.
- ke88y 3y ago
  I'm laughing at the study methodology, not the underlying topic. Please don't call me idiotic. I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.
  - arisAlexis 3y ago
    [flagged]
    - ke88y 3y ago
      > I repeat everyone that is loling at x-risk an idiot I mean this is the nicest possible way and hope that you consider it constructive: you're willfully misconstruing what I am saying and then calling me an idiot. This isn't a nice thing to do and shuts down the conversation. If I believed you were operating in good faith, I might take the time to explain why my practical/scientific experience makes me incredulous of what's technically possible, and why my life experience makes me suspicious of the personalities involved in pushing AI x-risk. I might also provide a history lesson on Einstein and Oppenheimer, which is instructive to how we should think about x-risk (spoiler: both advocated for and were involved in the development of atomic weapons). But since you're calling me an idiot and misconstruing my words, I have no interest in conversing with you. Have a good day.
      - vhlhvjcov 3y ago
        For those of us who don’t think you are an idiot (I don’t), could you maybe give us your insights and a history lesson? I am most intrigued, particularly regarding Oppenheimer and Einstein.
        ke88y 3y ago
        The tldr is that both of them urged Roosevelt to develop the weapon, and only later when the destructive potential of the bomb was obvious expressed regret. Einstein's 1938 letter to Roosevelt was the first step toward the Manhattan project. See https://www.osti.gov/opennet/manhattan-project-history/Resources/einstein_letter_photograph.htm#1 https://www.osti.gov/opennet/manhattan-project-history/Resou... if you want to read more. So it's weird to say Einstein "warned us" about the x-risk of nuclear weapons prior to their development when his letter to Roosevelt was begging for more money to speed up the development of nukes. I think the entire saga is mostly an unrelated red herring -- AI is nothing like nuclear bombs and reasoning by analogy in that way is sloppy at best. Mostly? It's just kind of funny that x-risk people point to Einstein and Oppenheimer as positive examples, since they both did literally the exact opposite of "warn the public and don't develop". The irony makes you chuckle if you know the history. Particularly given the weird fetish for IQ in the portion of the x-risk community that overlaps with the rationalist community, it's also really funny to point out that what they should be saying is actually something like "don't be like those high-IQ fools Einstein and Oppenheimer! They are terrible examples!" ;)
ALittleLight 3y ago
On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts in the field. One side seems to have much better quality of evidence. "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! What would you consider evidence of a significant AI risk? From my point of view (x-risk believer) the arguments in favor of existential risk are obvious and compelling. That many experts in the field agree seems like validation of my personal judgement of the arguments. Surveys of researchers likewise seem to confirm this. What evidence do you think is lacking from this side that would convince you?
- skepticATX 3y ago
  How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals? Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?
  - ke88y 3y ago
    >Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this? Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered. There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :) But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible. It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.
tiffanyg 3y ago
You may be far better informed than many you're arguing with, but casual dismissal IS foolish. The biggest potential / likely issues here aren't mere capabilities of systems and simple replacement of humans here and there. It's acceleration of "truth decay", accelerating and ever-more dramatic economic, social, and political upheaval, etc. You do not need "Terminator" for there to be dramatic downsides and damage from this technology. I'm no "doomer", but, looking at the bigger picture and considering upheavals that have occurred in the past, I am more convinced there's danger here than around any other revolutionary technologies I've seen break into the public consciousness and take off. Arguments about details, what's possible and what's not, limitations of systems, etc. - missing the forest for the trees IMO. I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world, vs. picking away at the small potatoes details-levels arguments...
- ke88y 3y ago
  I'm dismissing the survey, and I think you should as well. That's an orthogonal issue to actual x-risk, and I covered a bit more here: https://news.ycombinator.com/item?id=36577523 https://news.ycombinator.com/item?id=36577523 > I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world I think this is excellent advice and we're on the same page.
  - TedDoesntTalk 3y ago
    Where can I read RAND white papers on the topic?
    - ke88y 3y ago
      I think RAND was used here in the proverbial sense, not the literal sense. They generally do good work, stay away from hype while remaining relevant/timely, and focus on boring but real problems. Sort of that good spot in-between an IEEE Standards Committee and the press team for a VC/CEO/Influencer type. I think OP meant "that type of thing", rather than RAND per se. But in any case, there is https://www.rand.org/topics/artificial-intelligence.html https://www.rand.org/topics/artificial-intelligence.html
pseudonom- 3y ago
> Ie even lower than my guess of 90% above. These are not comparable numbers. You're comparing "fraction of people" vs "fraction of outcomes". Presumably an eye-roller assigns ~0 probability to "extremely bad" outcomes (or has a shockingly cavalier attitude toward medium-small probabilities of catastrophe).
- deleted 3y ago
  [deleted]
- ke88y 3y ago
  Yeah, that's correct. Stupid mistake on my part; was writing that comment in a hurry. Not sure why you're grayed out tbh. Thanks for the corrective. > or has a shockingly cavalier attitude Meh. Median response time was 40 seconds. The question didn't have a bounded time-frame for the risk. Five is small but non-zero. Also all of the other issues I've already pointed out. PhD students spending half a minute and writing down a number about risk over an unbounded time-frame is totally uninformative if you want to know how seriously experts take x-risk in time-frames that are relevant to any sort of policy or decision making. I think you and everyone else making comments about "shockingly cavalier attitude" wildly over-estimate the amount of thought and effort that respondents spend on this question. The "probability times magnitude" framing is not how normal people think about that question. I'd bet they just wrote down a small but not zero number; I'd probably write down 1 or 2 but definitely roll my eyes hard.

by ke88y 3y ago

arisAlexis 3y ago

Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.

ke88y 3y ago
I'm laughing at the study methodology, not the underlying topic. Please don't call me idiotic. I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.
- arisAlexis 3y ago
  [flagged]
  - ke88y 3y ago
    > I repeat everyone that is loling at x-risk an idiot I mean this is the nicest possible way and hope that you consider it constructive: you're willfully misconstruing what I am saying and then calling me an idiot. This isn't a nice thing to do and shuts down the conversation. If I believed you were operating in good faith, I might take the time to explain why my practical/scientific experience makes me incredulous of what's technically possible, and why my life experience makes me suspicious of the personalities involved in pushing AI x-risk. I might also provide a history lesson on Einstein and Oppenheimer, which is instructive to how we should think about x-risk (spoiler: both advocated for and were involved in the development of atomic weapons). But since you're calling me an idiot and misconstruing my words, I have no interest in conversing with you. Have a good day.
    - vhlhvjcov 3y ago
      For those of us who don’t think you are an idiot (I don’t), could you maybe give us your insights and a history lesson? I am most intrigued, particularly regarding Oppenheimer and Einstein.
      - ke88y 3y ago
        The tldr is that both of them urged Roosevelt to develop the weapon, and only later when the destructive potential of the bomb was obvious expressed regret. Einstein's 1938 letter to Roosevelt was the first step toward the Manhattan project. See https://www.osti.gov/opennet/manhattan-project-history/Resources/einstein_letter_photograph.htm#1 https://www.osti.gov/opennet/manhattan-project-history/Resou... if you want to read more. So it's weird to say Einstein "warned us" about the x-risk of nuclear weapons prior to their development when his letter to Roosevelt was begging for more money to speed up the development of nukes. I think the entire saga is mostly an unrelated red herring -- AI is nothing like nuclear bombs and reasoning by analogy in that way is sloppy at best. Mostly? It's just kind of funny that x-risk people point to Einstein and Oppenheimer as positive examples, since they both did literally the exact opposite of "warn the public and don't develop". The irony makes you chuckle if you know the history. Particularly given the weird fetish for IQ in the portion of the x-risk community that overlaps with the rationalist community, it's also really funny to point out that what they should be saying is actually something like "don't be like those high-IQ fools Einstein and Oppenheimer! They are terrible examples!" ;)

ALittleLight 3y ago

On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts in the field. One side seems to have much better quality of evidence. "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive! What would you consider evidence of a significant AI risk? From my point of view (x-risk believer) the arguments in favor of existential risk are obvious and compelling. That many experts in the field agree seems like validation of my personal judgement of the arguments. Surveys of researchers likewise seem to confirm this. What evidence do you think is lacking from this side that would convince you?

skepticATX 3y ago
How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals? Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?
- ke88y 3y ago
  >Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this? Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered. There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :) But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible. It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.

tiffanyg 3y ago

You may be far better informed than many you're arguing with, but casual dismissal IS foolish. The biggest potential / likely issues here aren't mere capabilities of systems and simple replacement of humans here and there. It's acceleration of "truth decay", accelerating and ever-more dramatic economic, social, and political upheaval, etc. You do not need "Terminator" for there to be dramatic downsides and damage from this technology. I'm no "doomer", but, looking at the bigger picture and considering upheavals that have occurred in the past, I am more convinced there's danger here than around any other revolutionary technologies I've seen break into the public consciousness and take off. Arguments about details, what's possible and what's not, limitations of systems, etc. - missing the forest for the trees IMO. I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world, vs. picking away at the small potatoes details-levels arguments...

ke88y 3y ago
I'm dismissing the survey, and I think you should as well. That's an orthogonal issue to actual x-risk, and I covered a bit more here: https://news.ycombinator.com/item?id=36577523 https://news.ycombinator.com/item?id=36577523 > I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world I think this is excellent advice and we're on the same page.
- TedDoesntTalk 3y ago
  Where can I read RAND white papers on the topic?
  - ke88y 3y ago
    I think RAND was used here in the proverbial sense, not the literal sense. They generally do good work, stay away from hype while remaining relevant/timely, and focus on boring but real problems. Sort of that good spot in-between an IEEE Standards Committee and the press team for a VC/CEO/Influencer type. I think OP meant "that type of thing", rather than RAND per se. But in any case, there is https://www.rand.org/topics/artificial-intelligence.html https://www.rand.org/topics/artificial-intelligence.html

pseudonom- 3y ago

> Ie even lower than my guess of 90% above. These are not comparable numbers. You're comparing "fraction of people" vs "fraction of outcomes". Presumably an eye-roller assigns ~0 probability to "extremely bad" outcomes (or has a shockingly cavalier attitude toward medium-small probabilities of catastrophe).

deleted 3y ago
[deleted]
ke88y 3y ago
Yeah, that's correct. Stupid mistake on my part; was writing that comment in a hurry. Not sure why you're grayed out tbh. Thanks for the corrective. > or has a shockingly cavalier attitude Meh. Median response time was 40 seconds. The question didn't have a bounded time-frame for the risk. Five is small but non-zero. Also all of the other issues I've already pointed out. PhD students spending half a minute and writing down a number about risk over an unbounded time-frame is totally uninformative if you want to know how seriously experts take x-risk in time-frames that are relevant to any sort of policy or decision making. I think you and everyone else making comments about "shockingly cavalier attitude" wildly over-estimate the amount of thought and effort that respondents spend on this question. The "probability times magnitude" framing is not how normal people think about that question. I'd bet they just wrote down a small but not zero number; I'd probably write down 1 or 2 but definitely roll my eyes hard.