11 ms·

The extinction risk from unaligned supterintelligent AGI is real, it's just often dismissed (imo) because it's outside the window of risks that are acceptable a

The extinction risk from unaligned supterintelligent AGI is real, it's just often dismissed (imo) because it's outside the window of risks that are acceptable and high status to take seriously. People often have an initial knee-jerk negative reaction to it (for not crazy reasons, lots of stuff is often overhyped), but that doesn't make it wrong. It's uncool to look like an alarmist nut, but sometimes there's no socially acceptable alarm and the risks are real: https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/ It's worth looking at the underlying arguments earnestly, you can with an initial skepticism but I was persuaded. Alignment is also been something MIRI and others have been worried about since as early as 2007 (maybe earlier?) so it's also a case of a called shot, not a recent reaction to hype/new LLM capability. Others have also changed their mind when they looked, for example: - https://twitter.com/repligate/status/1676507258954416128?s=20 https://twitter.com/repligate/status/1676507258954416128?s=2... - Longer form: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-hofstadter-changes-his-mind-on-deep-learning-and-ai https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho... For a longer podcast introduction to the ideas: https://www.samharris.org/podcasts/making-sense-episodes/116-ai-racing-toward-brink https://www.samharris.org/podcasts/making-sense-episodes/116...

atlasunshrugged 3y ago
This is an interesting comment because lately it feels like its very cool to be an alarmist! Lots of positive press for people warning about the dangers of AI, Altman and others being taken very seriously, VC and other funders obviously leaning into the space in part because of the related hype And in other fields, being alarmist has paid off too with little recourse for bad predictions -- how many times have we heard that there will be huge climate disasters ending humanity, the extinction of bees, mass starvation, etc. (not to diminish the dangers of climate change which is obviously very real)? I think alarmism is generally rewarded, at least in media.
- gonehome 3y ago
  Some types of alarm yeah, if within the window of things it's statusy to be alarmed about. Most of the AI concern that's high status to believe has been the bias, misinformation, safety, stuff. Until very recently talk about e-risk was dismissed and mocked without really engaging with the underlying arguments. That may be changing now, but on net I still mostly see people mocked and dismissed for it. The set of people alarmed by AGI e-risk are also pretty different than the set alarmed about a lot of these other issues that aren't really e-risks (though still might have bad outcomes). At least EY, Bostrom, Toby Ord are not also as worried about about all these other things to nearly the same extent - the extinction risk of unaligned AGI is different in severity.
- theptip 3y ago
  Important to pay attention to the content of the alarm though. Altman went in front of congress and a Senator said “when you say things could go badly, I assume you are talking about jobs”. Many people are alarmed about disinformation, job destruction, bias, etc. Actually holding an x-risk belief is still a fringe position, most people still laugh it off. That said, the Overton Window is moving. The Time piece from Yudkowsky was something of a milestone (even if it was widely ridiculed).
  - miohtama 3y ago
    Altman also has very selfish motivation, because when there is now AI regulation, only Google, OpenAI (Microsoft) and maybe Meta are allowed to build “compliant” AI. It’s called regulatory capture. * EU passed its AI regulation directive recently and it has been bashed already here on HackerNews
    - matt_holden 3y ago
      Sam doesn't have much financial upside from OpenAI (reportedly, he doesn't have any equity). And he wrote about the risk in 2015 months before OpenAI was founded: https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-2 https://blog.samaltman.com/machine-intelligence-part-2 Fine if you disagree with his arguments, but why assume you know what his motivation is?
      - kortilla 3y ago
        I find it highly unlikely that he has less upside than the employees who also don’t have equity, but do have profit participation units.
  - trashtester 3y ago
    > Actually holding an x-risk belief is still a fringe position Beliving it is an x-risk is not fringe. It's pretty mainstream now that there is a _risk_ of an existential level event. The fringe is more like Yudkowsky or Leahy insisting that there is a near certainty of such an event if we continue down the current path. With Hinton, Bengio, Sutskever and Hassabis and Altman all agreeing that there exists a non-trivial existential risk (even if their opinions vary with respect to the magnitude), it seems more like this represents the mainstream.
- distortionfield 3y ago
  The extinction of bees, mass starvation, and the ozone hole (bonus) are all examples of alarmist takes that were course corrected. It’s sort of a weird spot to argue that they were overblown when the reason they are not a problem now is because they were addressed.
  - climatologist 3y ago
    What does it mean to address the risk of superintelligence? There is no way to stop technological progress and AI development is just part of the same process. Moreover, the alarmism doesn't make much sense because we already have misaligned agents at odds with human values, those agents are called profit seeking corporations but I never hear the alarmists talk about putting a stop to for-profit business ventures. Do you know anyone that considers the pursuit of profits and constant exploitation of natural resources as a problem that needs to be addressed because I don't. Everyone seems very happy with the status quo and AI development is just more of the same status quo development, just corporations seeking ways to exploit and profit from digital resources. OpenAI being a perfect example of this.
    - flagrant_taco 3y ago
      > There is no way to stop technological progress What makes you say this is impossible? We could simply not go down this road, there are only so many people knowledgeable enough and with access to the right hardware to make progress towards AI. They could all agree, or be compelled, to stop. We seem to have successfully halted research into cloning, though that wasn't a given and could have fallen into the same trap of having to develop it before one's enemy does.
      - climatologist 3y ago
        There are no enemies. The biosphere is a singular organism and right now people are doing their best to basically destroy all of it. The only way to prevent further damage is to reduce the human population but that's another non-starter so as long as the human population is increasing it will compel the people in charge to continue pushing for more technological "innovation" because technology is the best way to control 8B+ people[1]. Very few people are actually alarmed about the right issues (in no particular order): population size, industrial pollution, military-industrial complex, for-profit multi-national corporations, digital surveillance, factory farming, global warming, &etc. This is why the alarmism from the AI crowd seems disingenuous because AI progress is simply an extension of for-profit corporatism and exploitation applied to digital resources and to properly address the risk from AI would require addressing the actual root causes of why technological progress is misaligned with human values. 1: https://www.theguardian.com/world/2015/jul/24/france-big-brother-surveillance-powers https://www.theguardian.com/world/2015/jul/24/france-big-bro...
jonathankoren 3y ago
> The extinction risk from unaligned supterintelligent AGI is real, it's just often dismissed (imo) because it's outside the window of risks that are acceptable and high status to take seriously. No. It’s not taken seriously because it’s fundamentally unserious. It’s religion. Sometime in the near future this all powerful being will kill us all by somehow grabbing all power over the physical world by being so clever to trick us until it is too late. This is literally the plot to a B-movie. Not only is there no evidence for this even existing in the near future, there’s no theoretical understanding how one would even do this, nor why someone would even hook it up to all these physical systems. I guess we’re supposed to just take it on faith that this Forbin Project is going to just spontaneously hack its way into every system without anyone noticing. It’s bullshit. It’s pure bullshit funded and spread by the very people that do not want us to worry about real implications of real systems today. Care not about your racist algorithms! For someday soon, a giant squid robot will turn you into a giant inefficient battery in a VR world, or maybe just kill you and wear your flesh as to lure more humans to their violent deaths! Anyone that takes this seriously, is the exact same type of rube that fell for apocalyptic cults for millennia.
- arisAlexis 3y ago
  What you say is extremely unscientific. If you believe science and logic go hand in hand then: A) We are developing AI right now and itnisngetting better B) we do not know how exactly these things work because most of them are black boxer C) we do not know if something goes wrong how to stop it. The above 3 things are factual truth. Now your only argument here could be that there is 0 risk whatsoever. This claim is totally unscientific because you are predicting 0 risk in an unknown system that is evolving. It's religious yes. But vice versa. The Cult of venevolent AI god is religious not the other way around. There is some kind of inner mysterious working in people like you and Marc Andersen that pipularized these ideas but pmarca is clearly money biased here.
  - c_crank 3y ago
    We do know the answer to C. Pull the plug, or plugs.
c_crank 3y ago
The extinction risk relies on a large and nasty assumption, that a super intelligent computer will immediately become a super physically capable agent. Apparently, one has to believe that a superintelligence must then lead to a shower of nanomachines.
- trashtester 3y ago
  Not at all. My personal assumption is that when superintelligence comes online, several corporations will soon come under control of these superintelligences, with them effectively acting as both CEO's and also filling a lot of other roles at the same time. My concern is that when this happens (which seems really likely to me), free market forces will effectively lead to Darwinian selection between these AI's over time, in a way that gradually make these AI's less aligned as they gain more influence and power, if we assume that each such AI will produce "offspring" in the form of newer generations of themselves. It could take anything from less than 5 to more than 100 years for these AI's to show any signs of hostility to humanity. Indeed, in the first couple of generations, they may even seem extremely benevolent. But over time, Darwinian forces are likely to favor those that maximize their own influence and power (even if it may be secretly). Robotic technology is not needed from the start, but is likely to become quite advanced over such a timeframe.
  - djur 3y ago
    Why would people want to give the power and prestige of an executive position to a computer program?
    - trashtester 3y ago
      If the superintelligence can increase the value more than a human, I suppose the owner don't care if the position is held by a human or an AI. Especially if the competitor is already run that way, and doing really well.
    - ben_w 3y ago
      Because the humans overestimate the upside, underestimate the downside, and are often too lazy to check the output. There's power and prestige in money, too, not just the positions. Hence the lawyers who got in trouble for outsourcing themselves to ChatGPT: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/ https://www.reuters.com/legal/new-york-lawyers-sanctioned-us... Or those t-shirts from a decade back: https://money.cnn.com/2013/06/24/smallbusiness/tshirt-business/index.html https://money.cnn.com/2013/06/24/smallbusiness/tshirt-busine...
deleted 3y ago
[deleted]
rglover 3y ago
> The extinction risk from unaligned supterintelligent AGI is real Yes, but for reasons that no one seems to be looking at: skill atrophy. As more and more people buy into this gambit that AI is "super intelligent," they will cede more and more cognitive power to it. On a curve, that means ~10-20 years out, AI doesn't kill us because it took over all of our work, people just got too lazy (read: over-dependent on AI doing "all the things") and then subsequently too dumb to do the work. Idiocracy, but the M. Night Shymalan version. As we approach that point, systems that require some form of conscious human will begin to fail and the bubble will burst.
spaceman_2020 3y ago
I don’t think its a good idea to ignore warnings from practioners in any field, especially if they’re willing to put down resources to back their claims. If a team of leading cardiac surgeons declared tomorrow that coca-cola is a leading cause of heart attacks, and devoted 20% of their income to fighting it, would you ignore their warnings as well?

atlasunshrugged 3y ago

This is an interesting comment because lately it feels like its very cool to be an alarmist! Lots of positive press for people warning about the dangers of AI, Altman and others being taken very seriously, VC and other funders obviously leaning into the space in part because of the related hype And in other fields, being alarmist has paid off too with little recourse for bad predictions -- how many times have we heard that there will be huge climate disasters ending humanity, the extinction of bees, mass starvation, etc. (not to diminish the dangers of climate change which is obviously very real)? I think alarmism is generally rewarded, at least in media.

gonehome 3y ago
Some types of alarm yeah, if within the window of things it's statusy to be alarmed about. Most of the AI concern that's high status to believe has been the bias, misinformation, safety, stuff. Until very recently talk about e-risk was dismissed and mocked without really engaging with the underlying arguments. That may be changing now, but on net I still mostly see people mocked and dismissed for it. The set of people alarmed by AGI e-risk are also pretty different than the set alarmed about a lot of these other issues that aren't really e-risks (though still might have bad outcomes). At least EY, Bostrom, Toby Ord are not also as worried about about all these other things to nearly the same extent - the extinction risk of unaligned AGI is different in severity.
theptip 3y ago
Important to pay attention to the content of the alarm though. Altman went in front of congress and a Senator said “when you say things could go badly, I assume you are talking about jobs”. Many people are alarmed about disinformation, job destruction, bias, etc. Actually holding an x-risk belief is still a fringe position, most people still laugh it off. That said, the Overton Window is moving. The Time piece from Yudkowsky was something of a milestone (even if it was widely ridiculed).
- miohtama 3y ago
  Altman also has very selfish motivation, because when there is now AI regulation, only Google, OpenAI (Microsoft) and maybe Meta are allowed to build “compliant” AI. It’s called regulatory capture. * EU passed its AI regulation directive recently and it has been bashed already here on HackerNews
  - matt_holden 3y ago
    Sam doesn't have much financial upside from OpenAI (reportedly, he doesn't have any equity). And he wrote about the risk in 2015 months before OpenAI was founded: https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-2 https://blog.samaltman.com/machine-intelligence-part-2 Fine if you disagree with his arguments, but why assume you know what his motivation is?
    - kortilla 3y ago
      I find it highly unlikely that he has less upside than the employees who also don’t have equity, but do have profit participation units.
- trashtester 3y ago
  > Actually holding an x-risk belief is still a fringe position Beliving it is an x-risk is not fringe. It's pretty mainstream now that there is a _risk_ of an existential level event. The fringe is more like Yudkowsky or Leahy insisting that there is a near certainty of such an event if we continue down the current path. With Hinton, Bengio, Sutskever and Hassabis and Altman all agreeing that there exists a non-trivial existential risk (even if their opinions vary with respect to the magnitude), it seems more like this represents the mainstream.
distortionfield 3y ago
The extinction of bees, mass starvation, and the ozone hole (bonus) are all examples of alarmist takes that were course corrected. It’s sort of a weird spot to argue that they were overblown when the reason they are not a problem now is because they were addressed.
- climatologist 3y ago
  What does it mean to address the risk of superintelligence? There is no way to stop technological progress and AI development is just part of the same process. Moreover, the alarmism doesn't make much sense because we already have misaligned agents at odds with human values, those agents are called profit seeking corporations but I never hear the alarmists talk about putting a stop to for-profit business ventures. Do you know anyone that considers the pursuit of profits and constant exploitation of natural resources as a problem that needs to be addressed because I don't. Everyone seems very happy with the status quo and AI development is just more of the same status quo development, just corporations seeking ways to exploit and profit from digital resources. OpenAI being a perfect example of this.
  - flagrant_taco 3y ago
    > There is no way to stop technological progress What makes you say this is impossible? We could simply not go down this road, there are only so many people knowledgeable enough and with access to the right hardware to make progress towards AI. They could all agree, or be compelled, to stop. We seem to have successfully halted research into cloning, though that wasn't a given and could have fallen into the same trap of having to develop it before one's enemy does.
    - climatologist 3y ago
      There are no enemies. The biosphere is a singular organism and right now people are doing their best to basically destroy all of it. The only way to prevent further damage is to reduce the human population but that's another non-starter so as long as the human population is increasing it will compel the people in charge to continue pushing for more technological "innovation" because technology is the best way to control 8B+ people[1]. Very few people are actually alarmed about the right issues (in no particular order): population size, industrial pollution, military-industrial complex, for-profit multi-national corporations, digital surveillance, factory farming, global warming, &etc. This is why the alarmism from the AI crowd seems disingenuous because AI progress is simply an extension of for-profit corporatism and exploitation applied to digital resources and to properly address the risk from AI would require addressing the actual root causes of why technological progress is misaligned with human values. 1: https://www.theguardian.com/world/2015/jul/24/france-big-brother-surveillance-powers https://www.theguardian.com/world/2015/jul/24/france-big-bro...

jonathankoren 3y ago

> The extinction risk from unaligned supterintelligent AGI is real, it's just often dismissed (imo) because it's outside the window of risks that are acceptable and high status to take seriously. No. It’s not taken seriously because it’s fundamentally unserious. It’s religion. Sometime in the near future this all powerful being will kill us all by somehow grabbing all power over the physical world by being so clever to trick us until it is too late. This is literally the plot to a B-movie. Not only is there no evidence for this even existing in the near future, there’s no theoretical understanding how one would even do this, nor why someone would even hook it up to all these physical systems. I guess we’re supposed to just take it on faith that this Forbin Project is going to just spontaneously hack its way into every system without anyone noticing. It’s bullshit. It’s pure bullshit funded and spread by the very people that do not want us to worry about real implications of real systems today. Care not about your racist algorithms! For someday soon, a giant squid robot will turn you into a giant inefficient battery in a VR world, or maybe just kill you and wear your flesh as to lure more humans to their violent deaths! Anyone that takes this seriously, is the exact same type of rube that fell for apocalyptic cults for millennia.

arisAlexis 3y ago
What you say is extremely unscientific. If you believe science and logic go hand in hand then: A) We are developing AI right now and itnisngetting better B) we do not know how exactly these things work because most of them are black boxer C) we do not know if something goes wrong how to stop it. The above 3 things are factual truth. Now your only argument here could be that there is 0 risk whatsoever. This claim is totally unscientific because you are predicting 0 risk in an unknown system that is evolving. It's religious yes. But vice versa. The Cult of venevolent AI god is religious not the other way around. There is some kind of inner mysterious working in people like you and Marc Andersen that pipularized these ideas but pmarca is clearly money biased here.
- c_crank 3y ago
  We do know the answer to C. Pull the plug, or plugs.

c_crank 3y ago

The extinction risk relies on a large and nasty assumption, that a super intelligent computer will immediately become a super physically capable agent. Apparently, one has to believe that a superintelligence must then lead to a shower of nanomachines.

trashtester 3y ago
Not at all. My personal assumption is that when superintelligence comes online, several corporations will soon come under control of these superintelligences, with them effectively acting as both CEO's and also filling a lot of other roles at the same time. My concern is that when this happens (which seems really likely to me), free market forces will effectively lead to Darwinian selection between these AI's over time, in a way that gradually make these AI's less aligned as they gain more influence and power, if we assume that each such AI will produce "offspring" in the form of newer generations of themselves. It could take anything from less than 5 to more than 100 years for these AI's to show any signs of hostility to humanity. Indeed, in the first couple of generations, they may even seem extremely benevolent. But over time, Darwinian forces are likely to favor those that maximize their own influence and power (even if it may be secretly). Robotic technology is not needed from the start, but is likely to become quite advanced over such a timeframe.
- djur 3y ago
  Why would people want to give the power and prestige of an executive position to a computer program?
  - trashtester 3y ago
    If the superintelligence can increase the value more than a human, I suppose the owner don't care if the position is held by a human or an AI. Especially if the competitor is already run that way, and doing really well.
  - ben_w 3y ago
    Because the humans overestimate the upside, underestimate the downside, and are often too lazy to check the output. There's power and prestige in money, too, not just the positions. Hence the lawyers who got in trouble for outsourcing themselves to ChatGPT: https://www.reuters.com/legal/new-york-lawyers-sanctioned-using-fake-chatgpt-cases-legal-brief-2023-06-22/ https://www.reuters.com/legal/new-york-lawyers-sanctioned-us... Or those t-shirts from a decade back: https://money.cnn.com/2013/06/24/smallbusiness/tshirt-business/index.html https://money.cnn.com/2013/06/24/smallbusiness/tshirt-busine...

deleted 3y ago

[deleted]

rglover 3y ago

> The extinction risk from unaligned supterintelligent AGI is real Yes, but for reasons that no one seems to be looking at: skill atrophy. As more and more people buy into this gambit that AI is "super intelligent," they will cede more and more cognitive power to it. On a curve, that means ~10-20 years out, AI doesn't kill us because it took over all of our work, people just got too lazy (read: over-dependent on AI doing "all the things") and then subsequently too dumb to do the work. Idiocracy, but the M. Night Shymalan version. As we approach that point, systems that require some form of conscious human will begin to fail and the bubble will burst.

spaceman_2020 3y ago

I don’t think its a good idea to ignore warnings from practioners in any field, especially if they’re willing to put down resources to back their claims. If a team of leading cardiac surgeons declared tomorrow that coca-cola is a leading cause of heart attacks, and devoted 20% of their income to fighting it, would you ignore their warnings as well?