30 ms·

My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think

My estimation of Eliezer Yudkowsky just dropped a notch — not because I don't take the existential threat posed by AI seriously — but because he seems to think that global coordination and control is an even remotely possible strategy.

concordDance 3y ago
He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.
- jeffparsons 3y ago
  It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
  - concordDance 3y ago
    That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.
    - jeffparsons 3y ago
      It depends on there being _some chance_ that it can be achieved. The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
      - TeMPOraL 3y ago
        Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
        varjag 3y ago
        Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
        TeMPOraL 3y ago
        I did, and you're right in both of your points.
  - johnthewise 3y ago
    He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U
    - jeffparsons 3y ago
      You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
    - hollerith 3y ago
      >He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
      - oceanplexian 3y ago
        I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
  - TeMPOraL 3y ago
    > Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal. From the way you put it just now: - "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone. - "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire. - "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war. > Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply. Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention. Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby. Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact. But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky! > If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. There are fates worse than death. Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident. > At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips. With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone". AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world. -- [0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.
- zimpenfish 3y ago
  > He's pretty sure that human civilization will be extinct this century. If they are, it'll almost certainly[1] be climate change or nuclear war, it won't be AI. [1] leaving some wiggle room for pandemics, asteroids, etc.
  - toth 3y ago
    There is no scenario where climate change leads to human extinction. Worst case scenario is global population goes down by a single digit percentage. Horrible, horrible stuff, but not extinction. And most likely scenarios are not nearly that bad. Nuclear war, if the worst nuclear winter predictions are correct, could be a lot worse but there would still be some survivors. Unaligned ASI though, could actually make us extinct.
    - bryanrasmussen 3y ago
      it said end of civilization? >Horrible, horrible stuff, but not extinction. anyway climate change drop single percentage directly caused, but that kind of thing seems like it would have LOTS of side effects.
    - kylebyte 3y ago
      I'm pretty sure the worst case of your nuclear war scenario is actually the worst case of the climate change one. Chunks of the world suddenly becoming unlivable and resources getting more scarce sounds like a recipe for escalation into war to me.
      - ethanbond 3y ago
        The nuclear war scenario, according to all known information about nuclear war policies, is not “chunks of the world becoming unlivable.” The principals in a nuclear conflict do not appear to even have a method to launch just a few nukes in response to a nuclear attacks: they will launch thousands of warheads at hundreds of cities.
        3y ago
- meowface 3y ago
  He's actually suggested it may happen within 10 years. Which makes it hard to take him seriously in general. (I do personally take his overall concerns seriously, but I don't trust his timeline predictions at all, and many of his other statements.)
  - kylebenzle 3y ago
    Can anyone enumerate what exactly your concerns are cause I don't get it. AI I'd a great spell checker, a REALLY good spell checker, and people are losing their minds over it, I don't get it.
    - ahamm 3y ago
      I used to think that this fear was driven by rational minds until I read Michael Lewis' "Going Infinite" and learned more about effective altruism [EA]. Michael Lewis writes that what is remarkable about these guys [i.e. EAs] is that they're willing to follow their beliefs to their logical conclusion (paraphrasing) without regard to social cost/consequences or just inconvenience of it all. In fact, in my estimation this is just the definition of religious fundamentalism, and gives us a new lens with which to understand EA and the well funded brain children of the movement. Every effective religion needs a doomsday scenario, or some 'second coming' apocalyptic like scenario (not sure why, just seems to be a pattern). I think all this fear mongering around AI is just that - it's based on irrational belief at the end of the day - its humanism rewashed into tech 'rationalism' (which was originally just washed from Christianity et. al.)
      - hm-nah 3y ago
        My comment above, which I think you’re replying to, is not based in fear. It’s based on first-hand experience experimenting with the likes of AutoExpert, AutoGen and ChatDev. The tools underlying these projects are quite close to doing, in a very short amount of time and cost, what it takes human knowledge workers a long time (and hence cost) to do. I think it’s as close as Summer 24’ that we get simplified GenAI grounding. Once hallucinations are grounded and there are some cleaner ways to implement GenAI workflows and pipelines…it won’t be long until you see droves of knowledge workers looking for jobs. Or if not, they’ll be creating the workflows that replace the bulk of their work, hence we’ll get that “I only want to work 20hrs a week” reality.
- tensor 3y ago
  If humans go extinct it sure as hell won't be AI that does it, but rather good old human ignorance and aggression.
  - sokoloff 3y ago
    I think the probability of humans eventually going extinct is 1. Avoiding it requires almost unimaginable levels of cooperation and technical advances.
    - boringg 3y ago
      Well thats not a difficult position to take. The difficult take is when the timing would be. The sun is eventually going to explode rendering our solar system uninhabitable though that is a long time away from now and we have many other massive risks to humanity in the meantime.
    - babyshake 3y ago
      The end of human civilization as we know it doesn't mean extinction. Why is extinction unavoidable?
      - TeMPOraL 3y ago
        Why does extinction matter? If we lose civilization now, we'll be stuck at pre-industrial levels possibly for geological timescales, as we've mined and burned all the high-density energy sources you can get at without already being a high-tech industrial society.
        ben_w 3y ago
        I don't think that would necessarily limit us. Windmills and water mills can be made with wood, stone, and the small quantities of iron that pre-industrial societies could produce.
        TeMPOraL 3y ago
- hodgesrm 3y ago
  > He's pretty sure that human civilization will be extinct this century. This is certainly possible, but if so extinction seems far more likely to come from nuclear warfare and/or disasters resulting from climate change. For example, it's become apparent that our food supply more brittle than many people. Losing access to fertilizers, running out of water, or disruptions in distribution netorks can lead to mass starvation by cutting the productivity we need to feed everyone.
  - civilitty 3y ago
    Is it really possible? Perhaps it's a failure of imagination but the only things I can think of that would guarantee human extinction are impacts from massive asteroids or nearby gamma ray burst or supernova. There are two billion poor subsistence farmers largely disconnected from market economies who aren't dependent on modern technology so the likelihood of them getting completely wiped out is very remote.
  - cjbprime 3y ago
    (Neither nuclear war nor climate change are considered as this-century extinction risks by the singularity-minded side of the existential risk community, just because there would necessarily be pockets of unaffected people.)
    - hodgesrm 3y ago
      I'm always a bit surprised at how people seem to wish away the destructiveness of nuclear weapons or the likelihood of their use. Another commenter alluded to a "failure of imagination" which seems very applicable here. You would think just watching videos of above-ground tests would cure that but apparently not. [0] [0] https://www.youtube.com/watch?v=gZuyPn0_KRE https://www.youtube.com/watch?v=gZuyPn0_KRE
      - cjbprime 3y ago
        I don't think I was doing either? Assume they are widely used. Does every human die?
- edgyquant 3y ago
  Why exactly should I care what this person thinks?
  - TremendousJudge 3y ago
    He wrote a really engaging Harry Potter fanfiction.
YetAnotherNick 3y ago
Global coordination using coercion has worked for nukes. It's very hard but not impossible to ban NVidia or any competitor and create spy network in all large companies and if any engineer is found to create AI, mark them as terrorist and give them POW treatment. And monitor chip fab like nuclear enrichment facility and launch war with country found to create chip fab, which I belive is hard to do it in complete secrecy. I believe if top 5 nation agrees for this, it could be acheived. Even with US and China enforcing this, it could likely be acheived.
- jeffparsons 3y ago
  Sure, I think that kind of "coordination" is a little more credible than what I thought Eliezer was proposing. Maybe I just didn't read carefully enough. One big difference here, however, is that the barrier to entry is lower. You can start doing meaningful AI R&D on existing hardware with _one_ clever person. The same was never true for nuclear weapons, so the pool of even potential players was pretty shallow for a long time, and therefore easier to control.
  - YetAnotherNick 3y ago
    > You can start doing meaningful AI R&D on existing hardware with _one_ clever person. With the current hardware, yes. But then again countries could highly restrict all decently powerful hardware and backdoor them, and you need permission from government to run matrix multiplication above few teraflops. Trying to mess with backdoor is a war crime. A weak form of hardware restriction even happens now that Nvidia couldn't export to many countries.
    - upupupandaway 3y ago
      And yet: https://www.scmp.com/news/china/science/article/3239998/chinese-scientists-create-chip-can-perform-ai-task-3000-times-faster-nvidias-a100-study https://www.scmp.com/news/china/science/article/3239998/chin...
  - johnthewise 3y ago
reducesuffering 3y ago
How are you anti-doomers arguments still this naive years later? No one is saying global coordination is in anyway remotely easy at all. They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic
- jeffparsons 3y ago
  I don't think that's a fair representation of my position. I'm not suggesting that it's "hard" — I'm suggesting that it's absolutely futile to attempt it, and at least as bad as doing nothing. I'm also suggesting that there are lots of other options remaining that are not necessarily catastrophic. I'm not particularly optimistic about any of them, but I think I've come to the opposite conclusion to you: I think that wasting effort on global coordination is getting is closer to that catastrophe. Whereas even the other options that involve unthinkably extreme violence might at least have some non-zero chance of working. I guess the irony of this reply is that I'm implying that I think _your_ position is naive. (Nuh-uh, you're the baby!) I suspect our main difference is that we have somehow formed very different beliefs about the likelihood of good/bad outcomes from different strategies. But I want to be very clear on one thing: I am not an "anti-doomer". I think there is a very strong probability that we are completely boned, no matter what mitigation strategies we might attempt.
- ImPostingOnHN 3y ago
  > They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic That's precisely what people have been saying about climate change. Some progress been made there, but if we can't solve that problem collectively (and we haven't been able to), we aren't going to be able to solve the "gain central control over AI" problem
varelse 3y ago
[dead]
SilasX 3y ago
But it will work fine for limiting GHG or CFC emissions?
- hanspeter 3y ago
  It will not. The difference is that reducing emissions just 10% is still better than 0% while stopping 90% from doing AI advancements is not better than 0% (it might actually be worse).
  - SilasX 3y ago
    That's a different argument than the parent, who claimed that global restriction isn't even possible.
    - hanspeter 3y ago
      I don't see that being different. I would also claim that global restriction is not possible and even in the unlikely scenario that we're able to restrict 90% of world, we're still not better off than if we did nothing.
      - SilasX 3y ago
        Still a different argument than one that offhand dismissed the very idea of global restriction at any level. Nothing you’re saying is found anywhere in the original comment. You have better uses of your time than retroactively rewriting others’ words to be less stupid so you can “refute” valid critiques of their original versions.

concordDance 3y ago

He actually doesn't think it's viable, but thinks its the only card left to play. He's pretty sure that human civilization will be extinct this century.

jeffparsons 3y ago
It can't be the only card left to play. There are so many others. They may be weak, but they still have to be better than one that is guaranteed to be absolutely useless. Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. Edit: or even try to convince _all_ governments to do this so that they come out on top. At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips.
- concordDance 3y ago
  That depends on making powerful AIs and being sure they are well aligned being as easy as making powerful AIs which look aligned to cursory inspection. The former seems much harder, though I'd say talk to Robert E. Miles for a more detailed look.
  - jeffparsons 3y ago
    It depends on there being _some chance_ that it can be achieved. The benchmark strategy we are comparing it against is tantamount to licking a wall repeatedly and expecting that to achieve something. So even a strategy that may well fail but has _some_ chance of success is a lot better by default.
    - TeMPOraL 3y ago
      Not with this. Failure to align a powerful AI is a mistake we only get to make once. This is like playing Russian roulette, where the amount of bullets missing from the magazine is the degree to which we can properly align the AI. Right now, we're at "the magazine is full, and we're betting our survival on the gun getting jammed".
      - varjag 3y ago
        Assuming you mean drum here; it's pointless to play russian roulette with a magazine fed weapon (which in a way could be an own metaphor for this predicament).
        TeMPOraL 3y ago
        I did, and you're right in both of your points.
- johnthewise 3y ago
  He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U
  - jeffparsons 3y ago
    You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
  - hollerith 3y ago
    >He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
    - oceanplexian 3y ago
      I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
- TeMPOraL 3y ago
  > Example of a weak, but at least _better_ option: convince _one_ government (e.g. USA) that it is in their interest to massively fund an effort to (1) develop an AI that is compliant to our wishes and can dominate all other AIs, and (2) actively sabotage competing efforts. I'd reconsider your revision of your estimate of Yudkowsky, as you seem to be dropping it for not proposing the very ideas he spent the last 20+ years criticizing, by explaining in every possible way how this is a) the default that's going to happen, b) dumb, and c) suicidal. From the way you put it just now: - "develop an AI that is compliant to our wishes" --> in other words, solving the alignment problem. Yes, this is the whole goal - and the reason Yudkowsky is calling for a moratorium on AI research enforced by a serious international treaty[0]. We still have little to no clue how to approach solving the problem, so an AI arms race now has a serious chance of birthing an AGI, which without the alignment problem being already solved, means game over for everyone. - "and can dominate all other AIs" --> short of building a self-improving AGI with ability to impose its will on other people (even if just in "enemy countries"), this will only fuel the AI arms race further. I can't see a version of this idea that ends better than just pressing the red button now, and burning the world in a nuclear fire. - "actively sabotage competing efforts" --> ah yes, this is how you turn an arms race into a hot war. > Governments have done much the same in the past with, e.g. conventional weapons, nuclear weapons, and cryptography, with varying levels of success. Any limit on conventional weapons that had any effect was backed by threat of bombing the living shit out of the violators. Otherwise, nations ignore them until they find a better/more effective alternative, after which it costs nothing to comply. Nuclear weapons are self-limiting. The first couple players locked the world in a MAD scenario, and now it's in everyone's best interest to not let anyone else have nukes. Also note that relevant treaties are, too, backed by threat of military intervention. Cryptography - this one was a bit dumb from the start, and ended up barely enforced. But note that where it is, the "or else bombs" card is always to be seen nearby. Can you see a pattern emerging? As I alluded to in the footnote [0] earlier, serious treaties always involve some form of "comply, or be bombed into compliance". Threat of war is always the final argument in international affairs, and you can tell how serious a treaty is by how directly it acknowledges that fact. But the ultimate point being: any success in the examples you listed was achieved exactly in the way Eliezer is proposing governments to act now. In that line, you're literally agreeing with Yudkowsky! > If we're all dead anyway otherwise, then I don't see how that can possibly be a worse card. There are fates worse than death. Think of factory farms, of the worst kind. The animals there would be better off dead than suffering through the things being done to them. Too bad they don't have that option - in fact, we proactively mutilate them so they can't kill themselves or each others, on purpose or in accident. > At least then there's a greater chance that the bleeding edge of this tech will be under the stewardship of a deliberate attempt for a country to dominate the world, rather than some bored kid who happens to stumble upon the recipe for global paperclips. With AI, there is no difference. The "use AI to dominate everyone else", besides sounding like a horrible dystopian future of the conventional kind, is just a tiny, tiny target to hit, next to a much larger target labeled "AI dominates everyone". AI risk isn't like nuclear weapons. It doesn't allow for a stable MAD state. It's more like engineered high-potency bioweapons - they start as more scary than effective, and continued refining turns them straight into a doomsday device. Continuing to develop them further only increases the chance of a lab accident suddenly ending the world. -- [0] - Yes, the "stop it, or else we bomb it to rubble" kind, because that is how international treaties look like when done by adults that care about the agreement being followed.
zimpenfish 3y ago
> He's pretty sure that human civilization will be extinct this century. If they are, it'll almost certainly[1] be climate change or nuclear war, it won't be AI. [1] leaving some wiggle room for pandemics, asteroids, etc.
- toth 3y ago
  There is no scenario where climate change leads to human extinction. Worst case scenario is global population goes down by a single digit percentage. Horrible, horrible stuff, but not extinction. And most likely scenarios are not nearly that bad. Nuclear war, if the worst nuclear winter predictions are correct, could be a lot worse but there would still be some survivors. Unaligned ASI though, could actually make us extinct.
  - bryanrasmussen 3y ago
    it said end of civilization? >Horrible, horrible stuff, but not extinction. anyway climate change drop single percentage directly caused, but that kind of thing seems like it would have LOTS of side effects.
  - kylebyte 3y ago
    I'm pretty sure the worst case of your nuclear war scenario is actually the worst case of the climate change one. Chunks of the world suddenly becoming unlivable and resources getting more scarce sounds like a recipe for escalation into war to me.
    - ethanbond 3y ago
      The nuclear war scenario, according to all known information about nuclear war policies, is not “chunks of the world becoming unlivable.” The principals in a nuclear conflict do not appear to even have a method to launch just a few nukes in response to a nuclear attacks: they will launch thousands of warheads at hundreds of cities.
      - 3y ago
meowface 3y ago
He's actually suggested it may happen within 10 years. Which makes it hard to take him seriously in general. (I do personally take his overall concerns seriously, but I don't trust his timeline predictions at all, and many of his other statements.)
- kylebenzle 3y ago
  Can anyone enumerate what exactly your concerns are cause I don't get it. AI I'd a great spell checker, a REALLY good spell checker, and people are losing their minds over it, I don't get it.
  - ahamm 3y ago
    I used to think that this fear was driven by rational minds until I read Michael Lewis' "Going Infinite" and learned more about effective altruism [EA]. Michael Lewis writes that what is remarkable about these guys [i.e. EAs] is that they're willing to follow their beliefs to their logical conclusion (paraphrasing) without regard to social cost/consequences or just inconvenience of it all. In fact, in my estimation this is just the definition of religious fundamentalism, and gives us a new lens with which to understand EA and the well funded brain children of the movement. Every effective religion needs a doomsday scenario, or some 'second coming' apocalyptic like scenario (not sure why, just seems to be a pattern). I think all this fear mongering around AI is just that - it's based on irrational belief at the end of the day - its humanism rewashed into tech 'rationalism' (which was originally just washed from Christianity et. al.)
    - hm-nah 3y ago
      My comment above, which I think you’re replying to, is not based in fear. It’s based on first-hand experience experimenting with the likes of AutoExpert, AutoGen and ChatDev. The tools underlying these projects are quite close to doing, in a very short amount of time and cost, what it takes human knowledge workers a long time (and hence cost) to do. I think it’s as close as Summer 24’ that we get simplified GenAI grounding. Once hallucinations are grounded and there are some cleaner ways to implement GenAI workflows and pipelines…it won’t be long until you see droves of knowledge workers looking for jobs. Or if not, they’ll be creating the workflows that replace the bulk of their work, hence we’ll get that “I only want to work 20hrs a week” reality.
tensor 3y ago
If humans go extinct it sure as hell won't be AI that does it, but rather good old human ignorance and aggression.
- sokoloff 3y ago
  I think the probability of humans eventually going extinct is 1. Avoiding it requires almost unimaginable levels of cooperation and technical advances.
  - boringg 3y ago
    Well thats not a difficult position to take. The difficult take is when the timing would be. The sun is eventually going to explode rendering our solar system uninhabitable though that is a long time away from now and we have many other massive risks to humanity in the meantime.
  - babyshake 3y ago
    The end of human civilization as we know it doesn't mean extinction. Why is extinction unavoidable?
    - TeMPOraL 3y ago
      Why does extinction matter? If we lose civilization now, we'll be stuck at pre-industrial levels possibly for geological timescales, as we've mined and burned all the high-density energy sources you can get at without already being a high-tech industrial society.
      - ben_w 3y ago
        I don't think that would necessarily limit us. Windmills and water mills can be made with wood, stone, and the small quantities of iron that pre-industrial societies could produce.
        TeMPOraL 3y ago
hodgesrm 3y ago
> He's pretty sure that human civilization will be extinct this century. This is certainly possible, but if so extinction seems far more likely to come from nuclear warfare and/or disasters resulting from climate change. For example, it's become apparent that our food supply more brittle than many people. Losing access to fertilizers, running out of water, or disruptions in distribution netorks can lead to mass starvation by cutting the productivity we need to feed everyone.
- civilitty 3y ago
  Is it really possible? Perhaps it's a failure of imagination but the only things I can think of that would guarantee human extinction are impacts from massive asteroids or nearby gamma ray burst or supernova. There are two billion poor subsistence farmers largely disconnected from market economies who aren't dependent on modern technology so the likelihood of them getting completely wiped out is very remote.
- cjbprime 3y ago
  (Neither nuclear war nor climate change are considered as this-century extinction risks by the singularity-minded side of the existential risk community, just because there would necessarily be pockets of unaffected people.)
  - hodgesrm 3y ago
    I'm always a bit surprised at how people seem to wish away the destructiveness of nuclear weapons or the likelihood of their use. Another commenter alluded to a "failure of imagination" which seems very applicable here. You would think just watching videos of above-ground tests would cure that but apparently not. [0] [0] https://www.youtube.com/watch?v=gZuyPn0_KRE https://www.youtube.com/watch?v=gZuyPn0_KRE
    - cjbprime 3y ago
      I don't think I was doing either? Assume they are widely used. Does every human die?
edgyquant 3y ago
Why exactly should I care what this person thinks?
- TremendousJudge 3y ago
  He wrote a really engaging Harry Potter fanfiction.

YetAnotherNick 3y ago

Global coordination using coercion has worked for nukes. It's very hard but not impossible to ban NVidia or any competitor and create spy network in all large companies and if any engineer is found to create AI, mark them as terrorist and give them POW treatment. And monitor chip fab like nuclear enrichment facility and launch war with country found to create chip fab, which I belive is hard to do it in complete secrecy. I believe if top 5 nation agrees for this, it could be acheived. Even with US and China enforcing this, it could likely be acheived.

jeffparsons 3y ago
Sure, I think that kind of "coordination" is a little more credible than what I thought Eliezer was proposing. Maybe I just didn't read carefully enough. One big difference here, however, is that the barrier to entry is lower. You can start doing meaningful AI R&D on existing hardware with _one_ clever person. The same was never true for nuclear weapons, so the pool of even potential players was pretty shallow for a long time, and therefore easier to control.
- YetAnotherNick 3y ago
  > You can start doing meaningful AI R&D on existing hardware with _one_ clever person. With the current hardware, yes. But then again countries could highly restrict all decently powerful hardware and backdoor them, and you need permission from government to run matrix multiplication above few teraflops. Trying to mess with backdoor is a war crime. A weak form of hardware restriction even happens now that Nvidia couldn't export to many countries.
  - upupupandaway 3y ago
    And yet: https://www.scmp.com/news/china/science/article/3239998/chinese-scientists-create-chip-can-perform-ai-task-3000-times-faster-nvidias-a100-study https://www.scmp.com/news/china/science/article/3239998/chin...
- johnthewise 3y ago

reducesuffering 3y ago

How are you anti-doomers arguments still this naive years later? No one is saying global coordination is in anyway remotely easy at all. They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic

jeffparsons 3y ago
I don't think that's a fair representation of my position. I'm not suggesting that it's "hard" — I'm suggesting that it's absolutely futile to attempt it, and at least as bad as doing nothing. I'm also suggesting that there are lots of other options remaining that are not necessarily catastrophic. I'm not particularly optimistic about any of them, but I think I've come to the opposite conclusion to you: I think that wasting effort on global coordination is getting is closer to that catastrophe. Whereas even the other options that involve unthinkably extreme violence might at least have some non-zero chance of working. I guess the irony of this reply is that I'm implying that I think _your_ position is naive. (Nuh-uh, you're the baby!) I suspect our main difference is that we have somehow formed very different beliefs about the likelihood of good/bad outcomes from different strategies. But I want to be very clear on one thing: I am not an "anti-doomer". I think there is a very strong probability that we are completely boned, no matter what mitigation strategies we might attempt.
ImPostingOnHN 3y ago
> They’re saying that it doesn’t matter what the difficulty is, the other option is fucking catastrophic That's precisely what people have been saying about climate change. Some progress been made there, but if we can't solve that problem collectively (and we haven't been able to), we aren't going to be able to solve the "gain central control over AI" problem

varelse 3y ago

[dead]

SilasX 3y ago

But it will work fine for limiting GHG or CFC emissions?

hanspeter 3y ago
It will not. The difference is that reducing emissions just 10% is still better than 0% while stopping 90% from doing AI advancements is not better than 0% (it might actually be worse).
- SilasX 3y ago
  That's a different argument than the parent, who claimed that global restriction isn't even possible.
  - hanspeter 3y ago
    I don't see that being different. I would also claim that global restriction is not possible and even in the unlikely scenario that we're able to restrict 90% of world, we're still not better off than if we did nothing.
    - SilasX 3y ago
      Still a different argument than one that offhand dismissed the very idea of global restriction at any level. Nothing you’re saying is found anywhere in the original comment. You have better uses of your time than retroactively rewriting others’ words to be less stupid so you can “refute” valid critiques of their original versions.