8 ms·

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_s

That's not what this survey shows: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...

echelon 3y ago
If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand. Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes. As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened. Look at their actions, not their words.
- zone411 3y ago
  If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
  - echelon 3y ago
    > If you read the survey Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer. > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it. > So, this type of response wouldn't be rational. This is the rational response to a 10% chance? These are funny numbers and nobody really has their skin in the game here. If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI. > especially if research continues in places like China. I like how both sides of this argument are using the specter of China as a means to further their argument.
    - phillipcarter 3y ago
      > Not a nice way to engage in debate. So...you didn't read the survey.
    - ALittleLight 3y ago
      What if some of the experts said "It's really hard to know, but our best guess is there is a 10% chance of this thing hitting the Earth" and other experts said "I really don't think it'll hit the Earth"? My best guess is that Earth wouldn't do much at all about the risk as that seems to be basically what we are facing with AI x-risk.
    - zone411 3y ago
      Yudkowsky represents the extreme end of risk concern. You can't fault others who estimate the risk at 10%, with huge uncertainty about when this risk will materialize and its chances, for not behaving as you'd expect him to. Believing that people will take extreme actions, which would ruin their careers and likely backfire, based on a 10% chance of things going terribly wrong in maybe 30 years is strange.
    - kalkin 3y ago
      "If [people who I disagree with] really believed what they said, they'd do [X Y or Z extreme thing]" is almost always a bad argument. Consider the possibility that instead, the people who actually believe the thing you don't have spent more time thinking seriously about their strategy than you have, and you are not in fact better both at being you and at being them than they are. I'm used to this pattern showing up as "lol if you really don't like capitalism how come you use money" but it's just as bad here.
      - mcpackieh 3y ago
        A less-than-optimal response isn't very indicative of anything, but I think you should definitely think twice when the intensity of the protest is grossly out of proportion with the intensity of the rhetoric. X-risk people are talking about the complete extermination of humanity but all they do is write internet essays about it. They aren't even availing themselves of standard protesting tactics, like standing outside AI businesses with signs or trying to intimidate researchers. Some form of real protest is table stakes for being taken seriously when you're crying about the end of the world.
- JoshTko 3y ago
  Is this a fair test? If you are a person with average person resources and don't expect you can impact what gets built why would you jeopardize your livelihood to make no impact?
- hackinthebochs 3y ago
  Oh look, an unfalsifiable claim in service to your predetermined position. How novel. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".
  - echelon 3y ago
    This whole thing is Pascal's wager. Damned if you do, damned if you don't. Nothing is falsifiable from your side either. The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus. These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that. Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway. If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.
    - hackinthebochs 3y ago
      There's something to be said for not going full steam ahead when we don't have a strong idea of the outcome. This idea that progress is an intrinsic good therefore we must continue the march of technology is a fallacy. It is extreme hubris to think that we will be able to control a potential superintelligence. The cost of being wrong is hard to overstate. >These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that. This doesn't track with the lessons learned from LLMs. The obscene amounts of compute thrown at modern networks changes the calculus completely. ChatGPT essentially existed for years in the form of GPT-3, but no one knew what they had. The lesson to learn is that capabilities can far outpace expectations when obscene amounts of computation are in play. >The people trying to regulate AI are concentrating economic upside into a handful of companies. Yes, its clear this is the motivations for much of the anti-doom folks. They don't want to be left out of the fun and profit. Their argument is downstream from this. No, doing things public isn't the answer to safety, just like doing bioengineering research or nuclear research in public isn't the answer to safety.
- yanderekko 3y ago
  >If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. I think we underestimate the intoxicating lure of human complacency at our own peril. If I think there's a 90% chance that AI will kill me in the next 20 years, maybe I'd be doing this. Of course, there is a knowing perception that appearing too unhinged can be detrimental to the cause, eg. actually instigating terrorist attacks against AI research labs may backfire. But if I only think there's a 25% chance? Ehh. My life will be less-stressed if I don't think about it too much, and just go on as normal. I'm going to die eventually anyways, if it's part of a singularity event then I imagine it will be quick and not too painful. Of course, if the 25% estimate were accurate, then it's by far the most important policy issue of the current day. Also of course there are collective action problems. If I think there's a 90% chance AI will kill me, then do I really think I can bring that down appreciably? Probably not. I could probably still have a bigger positive impact on my life expectancy by say dieting better. And let's see how good humans are at that..
- arisAlexis 3y ago
  "If these people were so concerned, they'd by shouting from the hilltops" they are, one by one
- api 3y ago
  > Eliezer is one of a handful of people putting their reputation on the line I have a hard time understanding why anyone takes Yudkowski seriously. What has he done other than found a cult around self-referential ideologies? By self-referential I mean the ideology's proof rests on its own claims and assertions. Rationalism is rational because it is rational according to its own assertions and methods, not because it has accomplished anything in the real world or been validated in any scientific or empirical-historical way. Longtermism is particularly inane. It defers everything to a hypothetical ~infinitely large value ~infinitely far in the future, thereby devaluing any real-world pragmatic problems that exist today. War? Climate change? Inequality? Refugee crises? None of that's important compared to the creation of trillions of hypothetical future minds in a hypothetical future utopia whose likelihood we can hypothetically maximize with NaN probability. You can see how absurd this is by applying it recursively. Let's say we have colonized the galaxy and there are in fact trillions of superintelligent minds living in a pain-free immortal near-utopia. Can we worry about mere proximate problems now? No, of course not. There are countless trillions of galaxies waiting to be colonized! Value is always deferred to a future beyond any living person's time horizon. The end result of this line of reasoning is the same as medieval religious scholasticism that deferred all questions of human well being to the next world. I just brought this up to provide one example of the inane nonsense this cult churns out. But what do I know. I obviously have a lower IQ than these people.
  - staunton 3y ago
    > found a cult around self-referential ideologies That's something. > Rationalism is rational because it is rational... In his ideology, "rational" means "the way of thinking that best lets you achieve your goals". This not self-referential. A more appropriate criticism might be "meaningless by itself". I guess the self-referential aspect is that you're supposed to think about whether or not you're thinking well. At a basic level, that sounds useful despite being self-referential, in the same way that bootstrapping is useful. The question is if course what Yudkowski makes of this basic premise, which is hard to evaluate. The controversy about "longtermism" has two parts. The first is a disagreement about how much to discount the future. Some people think "making absolutely sure humanity survives the next 1000 years" is very important, some people think it's not that important. There's really no way to settle this question, it's a matter of preference. The second part is about the actual estimate of how big some dangers are. The boring part of this is that people disagree about facts and models, where more discussion is the way to go if you care about the results (which you might not). However, there is a more interesting difference between people who are/aren't sympathetic to longtermism, which lies in how they think about uncertainty. For example, suppose you make your best possible effort (maybe someone paid you to make this worthwhile for you) to predict how likely some danger is. This prediction is now your honest opinion about this likelihood because if you'd thought it was over/under-estimated, you'd adjusted your model. Suppose also that your model seems very likely to be bad. You just don't know in which direction. In this situation, people sympathetic towards longtermism tend to say "that's my best prediction, it says there is a significant risk, we have to care about it. Let's take some precautions already and keep working on the model.". People who don't like it, in the same situation, tend to say "this model is probably wrong and hence tells us nothing useful. We shouldn't take precautions and stop modeling because it doesn't seem feasible to build a good model.". I think both sides have a point. One side would think and act as best they can, and take precautions against a big risk that's hard to evaluate. The other would prioritize actions that are likely useful and avoid spending resources on modeling if that's unlikely to lead to good predictions. I find it a very interesting question which of these ways of dealing with uncertainty is more appropriate in everyday life, or in some given circumstances . As you rightfully point out, the "rationalist/longtermmist" side of the discussion has an inherent tendency to detach from reality and lose itself discussing the details of lots of very unrealistic scenarios, which they must work hard to counteract. The ideas naturally attract people who enjoy armchair philosophizing and aren't likely to act based on concrete consequences of their abstract framework.
- hollerith 3y ago
  We are serious about stopping you. We judge that at the present time, the main hope is that the US and British governments will ban large training runs and hopefully also shut down most or all of the labs. We judge that unlawful actions like torching data centers make it less likely we will realize the main hope. Chinese society is much more likely to suddenly descend into chaos than the societies most of the people reading this are most familiar with, to briefly address a common objection on this site to the idea that a ban imposed by only the US and Britain will do any good. (It would be nice if there were some way to stop the reckless AI research done in China as well as that done in the US and Britain, but the fact that we probably cannot achieve that will not discourage us from trying for more achievable outcomes such as a ban in the US or Britain. I am much more worried about US and British AI research labs than I am of China's getting too powerful.)
- mquander 3y ago
  What do you mean, you haven't seen anything else like it? At least O(100) people are working full-time on research programs trying to mitigate AI x-risks. The whole founding team of Anthropic left OpenAI to do their own thing specifically because they thought that's what they needed to do to pursue their safety agenda. Isn't that taking the problem at least as seriously as quitting a job at Google? It sounds like you think that the main way to act on a concern is by making a maximum amount of noise about it. But another way to act on a concern is to try to solve it. Up until very recently, the population of people who perceive the risk are mostly engineers and scientists, not politicians or journalists, so they are naturally inclined towards the latter approach. In the end, if people aren't putting their money (really or metaphorically) where their mouth is, you can accuse them of not really caring, and if people are putting their money where their mouth is, then you can accuse them of just talking their book. So reasoning from whether they are acting exactly how you think they should, is not going to be a good way to figure out how right they are or aren't.
ke88y 3y ago
> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate. Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make. Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder. I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement. But even within that sample: > The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. Ie even lower than my guess of 90% above.
- arisAlexis 3y ago
  Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.
  - ke88y 3y ago
    I'm laughing at the study methodology, not the underlying topic. Please don't call me idiotic. I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.
    - arisAlexis 3y ago
      [flagged]
biggoodwolf 3y ago
Read the parent comment again, even taking the survey at face value: "EXPERTS DECLARE EXPERTS' FIELD IS MOST IMPORTATN!!!!" No news, only snooze

echelon 3y ago

If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand. Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes. As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened. Look at their actions, not their words.

zone411 3y ago
If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
- echelon 3y ago
  > If you read the survey Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer. > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it. > So, this type of response wouldn't be rational. This is the rational response to a 10% chance? These are funny numbers and nobody really has their skin in the game here. If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI. > especially if research continues in places like China. I like how both sides of this argument are using the specter of China as a means to further their argument.
  - phillipcarter 3y ago
    > Not a nice way to engage in debate. So...you didn't read the survey.
  - ALittleLight 3y ago
    What if some of the experts said "It's really hard to know, but our best guess is there is a 10% chance of this thing hitting the Earth" and other experts said "I really don't think it'll hit the Earth"? My best guess is that Earth wouldn't do much at all about the risk as that seems to be basically what we are facing with AI x-risk.
  - zone411 3y ago
    Yudkowsky represents the extreme end of risk concern. You can't fault others who estimate the risk at 10%, with huge uncertainty about when this risk will materialize and its chances, for not behaving as you'd expect him to. Believing that people will take extreme actions, which would ruin their careers and likely backfire, based on a 10% chance of things going terribly wrong in maybe 30 years is strange.
  - kalkin 3y ago
    "If [people who I disagree with] really believed what they said, they'd do [X Y or Z extreme thing]" is almost always a bad argument. Consider the possibility that instead, the people who actually believe the thing you don't have spent more time thinking seriously about their strategy than you have, and you are not in fact better both at being you and at being them than they are. I'm used to this pattern showing up as "lol if you really don't like capitalism how come you use money" but it's just as bad here.
    - mcpackieh 3y ago
      A less-than-optimal response isn't very indicative of anything, but I think you should definitely think twice when the intensity of the protest is grossly out of proportion with the intensity of the rhetoric. X-risk people are talking about the complete extermination of humanity but all they do is write internet essays about it. They aren't even availing themselves of standard protesting tactics, like standing outside AI businesses with signs or trying to intimidate researchers. Some form of real protest is table stakes for being taken seriously when you're crying about the end of the world.
JoshTko 3y ago
Is this a fair test? If you are a person with average person resources and don't expect you can impact what gets built why would you jeopardize your livelihood to make no impact?
hackinthebochs 3y ago
Oh look, an unfalsifiable claim in service to your predetermined position. How novel. You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".
- echelon 3y ago
  This whole thing is Pascal's wager. Damned if you do, damned if you don't. Nothing is falsifiable from your side either. The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus. These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that. Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway. If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.
  - hackinthebochs 3y ago
    There's something to be said for not going full steam ahead when we don't have a strong idea of the outcome. This idea that progress is an intrinsic good therefore we must continue the march of technology is a fallacy. It is extreme hubris to think that we will be able to control a potential superintelligence. The cost of being wrong is hard to overstate. >These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that. This doesn't track with the lessons learned from LLMs. The obscene amounts of compute thrown at modern networks changes the calculus completely. ChatGPT essentially existed for years in the form of GPT-3, but no one knew what they had. The lesson to learn is that capabilities can far outpace expectations when obscene amounts of computation are in play. >The people trying to regulate AI are concentrating economic upside into a handful of companies. Yes, its clear this is the motivations for much of the anti-doom folks. They don't want to be left out of the fun and profit. Their argument is downstream from this. No, doing things public isn't the answer to safety, just like doing bioengineering research or nuclear research in public isn't the answer to safety.
yanderekko 3y ago
>If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers. I think we underestimate the intoxicating lure of human complacency at our own peril. If I think there's a 90% chance that AI will kill me in the next 20 years, maybe I'd be doing this. Of course, there is a knowing perception that appearing too unhinged can be detrimental to the cause, eg. actually instigating terrorist attacks against AI research labs may backfire. But if I only think there's a 25% chance? Ehh. My life will be less-stressed if I don't think about it too much, and just go on as normal. I'm going to die eventually anyways, if it's part of a singularity event then I imagine it will be quick and not too painful. Of course, if the 25% estimate were accurate, then it's by far the most important policy issue of the current day. Also of course there are collective action problems. If I think there's a 90% chance AI will kill me, then do I really think I can bring that down appreciably? Probably not. I could probably still have a bigger positive impact on my life expectancy by say dieting better. And let's see how good humans are at that..
arisAlexis 3y ago
"If these people were so concerned, they'd by shouting from the hilltops" they are, one by one
api 3y ago
> Eliezer is one of a handful of people putting their reputation on the line I have a hard time understanding why anyone takes Yudkowski seriously. What has he done other than found a cult around self-referential ideologies? By self-referential I mean the ideology's proof rests on its own claims and assertions. Rationalism is rational because it is rational according to its own assertions and methods, not because it has accomplished anything in the real world or been validated in any scientific or empirical-historical way. Longtermism is particularly inane. It defers everything to a hypothetical ~infinitely large value ~infinitely far in the future, thereby devaluing any real-world pragmatic problems that exist today. War? Climate change? Inequality? Refugee crises? None of that's important compared to the creation of trillions of hypothetical future minds in a hypothetical future utopia whose likelihood we can hypothetically maximize with NaN probability. You can see how absurd this is by applying it recursively. Let's say we have colonized the galaxy and there are in fact trillions of superintelligent minds living in a pain-free immortal near-utopia. Can we worry about mere proximate problems now? No, of course not. There are countless trillions of galaxies waiting to be colonized! Value is always deferred to a future beyond any living person's time horizon. The end result of this line of reasoning is the same as medieval religious scholasticism that deferred all questions of human well being to the next world. I just brought this up to provide one example of the inane nonsense this cult churns out. But what do I know. I obviously have a lower IQ than these people.
- staunton 3y ago
  > found a cult around self-referential ideologies That's something. > Rationalism is rational because it is rational... In his ideology, "rational" means "the way of thinking that best lets you achieve your goals". This not self-referential. A more appropriate criticism might be "meaningless by itself". I guess the self-referential aspect is that you're supposed to think about whether or not you're thinking well. At a basic level, that sounds useful despite being self-referential, in the same way that bootstrapping is useful. The question is if course what Yudkowski makes of this basic premise, which is hard to evaluate. The controversy about "longtermism" has two parts. The first is a disagreement about how much to discount the future. Some people think "making absolutely sure humanity survives the next 1000 years" is very important, some people think it's not that important. There's really no way to settle this question, it's a matter of preference. The second part is about the actual estimate of how big some dangers are. The boring part of this is that people disagree about facts and models, where more discussion is the way to go if you care about the results (which you might not). However, there is a more interesting difference between people who are/aren't sympathetic to longtermism, which lies in how they think about uncertainty. For example, suppose you make your best possible effort (maybe someone paid you to make this worthwhile for you) to predict how likely some danger is. This prediction is now your honest opinion about this likelihood because if you'd thought it was over/under-estimated, you'd adjusted your model. Suppose also that your model seems very likely to be bad. You just don't know in which direction. In this situation, people sympathetic towards longtermism tend to say "that's my best prediction, it says there is a significant risk, we have to care about it. Let's take some precautions already and keep working on the model.". People who don't like it, in the same situation, tend to say "this model is probably wrong and hence tells us nothing useful. We shouldn't take precautions and stop modeling because it doesn't seem feasible to build a good model.". I think both sides have a point. One side would think and act as best they can, and take precautions against a big risk that's hard to evaluate. The other would prioritize actions that are likely useful and avoid spending resources on modeling if that's unlikely to lead to good predictions. I find it a very interesting question which of these ways of dealing with uncertainty is more appropriate in everyday life, or in some given circumstances . As you rightfully point out, the "rationalist/longtermmist" side of the discussion has an inherent tendency to detach from reality and lose itself discussing the details of lots of very unrealistic scenarios, which they must work hard to counteract. The ideas naturally attract people who enjoy armchair philosophizing and aren't likely to act based on concrete consequences of their abstract framework.
hollerith 3y ago
We are serious about stopping you. We judge that at the present time, the main hope is that the US and British governments will ban large training runs and hopefully also shut down most or all of the labs. We judge that unlawful actions like torching data centers make it less likely we will realize the main hope. Chinese society is much more likely to suddenly descend into chaos than the societies most of the people reading this are most familiar with, to briefly address a common objection on this site to the idea that a ban imposed by only the US and Britain will do any good. (It would be nice if there were some way to stop the reckless AI research done in China as well as that done in the US and Britain, but the fact that we probably cannot achieve that will not discourage us from trying for more achievable outcomes such as a ban in the US or Britain. I am much more worried about US and British AI research labs than I am of China's getting too powerful.)
mquander 3y ago
What do you mean, you haven't seen anything else like it? At least O(100) people are working full-time on research programs trying to mitigate AI x-risks. The whole founding team of Anthropic left OpenAI to do their own thing specifically because they thought that's what they needed to do to pursue their safety agenda. Isn't that taking the problem at least as seriously as quitting a job at Google? It sounds like you think that the main way to act on a concern is by making a maximum amount of noise about it. But another way to act on a concern is to try to solve it. Up until very recently, the population of people who perceive the risk are mostly engineers and scientists, not politicians or journalists, so they are naturally inclined towards the latter approach. In the end, if people aren't putting their money (really or metaphorically) where their mouth is, you can accuse them of not really caring, and if people are putting their money where their mouth is, then you can accuse them of just talking their book. So reasoning from whether they are acting exactly how you think they should, is not going to be a good way to figure out how right they are or aren't.

ke88y 3y ago

> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate. Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make. Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder. I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement. But even within that sample: > The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%. Ie even lower than my guess of 90% above.

arisAlexis 3y ago
Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.
- ke88y 3y ago
  I'm laughing at the study methodology, not the underlying topic. Please don't call me idiotic. I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.
  - arisAlexis 3y ago
    [flagged]

biggoodwolf 3y ago

Read the parent comment again, even taking the survey at face value: "EXPERTS DECLARE EXPERTS' FIELD IS MOST IMPORTATN!!!!" No news, only snooze