7 ms·

The Existential Risk of AGI

18 points by a_wild_dandan 3y ago

sva_ 3y ago
Difficult to read Joscha's post without hearing his voice in my head.
Zetobal 3y ago
As soon as chatgpt was released every hard scifi fan got a boner and a podcast interview invitation.
- grandma_tea 3y ago
  Dr Joscha Bach has been studying cognitive science and AI for the past 23 years.
  - ubu7737 3y ago
    [dead]
aaron695 3y ago
[dead]
29athrowaway 3y ago
The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.
- bilsbie 3y ago
  On the other hand self checkout still hasn’t replaced cashiers after 30 years.
  - deleted 3y ago
    [deleted]
  - acdha 3y ago
    Not entirely but it’s cut a lot of jobs and put downward pressure on wages. It doesn’t have to be a complete replacement to negatively impact lives.
  - andrewflnr 3y ago
    Uh, it's getting pretty close. At places that have it, I use it almost exclusively. They need human tech support, and security, but the basic checkout task is pretty well solved.
    - austinjp 3y ago
      "Solved" seems an odd way to view self-checkout. I find it a hideous experience. Everyone is irritable, staff and shoppers. The handful of staff have fewer tasks that involve any degree of skill, so they are reduced to simply keeping the machines working. Any issue is addressed just pressing the relevant button combo to resume regular function. If it can't be solved, maybe just close the till and move the customer to a different machine. Perhaps in busy periods the job might involve chivvying customers to available machines while calling "cash or card? Cash or card?" Customers are either bored or frustrated. Beeping your own shopping is like doing unpaid work, without even the distraction of some inane chat with a checkout jockey. And don't forget to tap your loyalty card so you can get ripped off with crumbs that are worth far less than the data you're handing over. I actively avoid the whole degrading, isolating circus. It's not "solved", quite the opposite: it's ripe for 'disruption'. Bah :)
      - ubu7737 3y ago
        [dead]
      - danaris 3y ago
        Do note that this experience is not universal. Where I see self-checkouts, I see customers choosing them voluntarily, and staff who seem perfectly cheerful and willing to help with the normal issues that arise, as well as "training" less-savvy customers to use them. Personally, I would use them every time if I could. I'd much rather minimize my interactions with strangers while shopping.
        austinjp 3y ago
        > Where I see self-checkouts, I see customers choosing them voluntarily Yeah there are confounding factors at play here, such as there being many automated checkouts but only one with a person. > staff ... "training" less-savvy customers to use them. Training the customers is one of the things I find most depressing. From the supermarket's perspective, customers have apparently accepted that they must do unpaid work that used to be done by paid staff. Scanning items, typing in product codes when the scanner doesn't work, looking up loose goods in a catalogue and weighing them, responding to beeps and prompts, etc. And even better, watching advertising. Sure, these are all 'first world problems', but societies should be improving their citizens' lives rather than whittling away their autonomy. It might be just a papercut, but it's death by a billion papercuts. > I'd much rather minimize my interactions with strangers while shopping. Social isolation is a major and growing problem in western societies (see my other comment). People choose the path of least resistance even if it's long-term suboptimal, and minimising interactions with strangers seems like an unfortunate choice. Obviously, I'm making a bunch of assumptions that might not be fair, and you have my sympathies if your neighbourhood is full of arseholes :)
- breuleux 3y ago
  I don’t know about that. I think AGI with 60 IQ may arrive before AGI that can walk, which rather limits its usefulness as an army.
i_like_apis 3y ago
Speculating about existential risk is certainly important, but way too easy to overstate. It can easily become distracting and actually dangerous in itself. Here’s a good take on it: https://www.ai-breakout.com/post/ai-alignment-and-the-messiah-bias https://www.ai-breakout.com/post/ai-alignment-and-the-messia... I also think we would all be very wise to remember the story of Henny Penny / “Chicken Little”. https://americanliterature.com/childrens-stories/henny-penny-the-sky-is-falling https://americanliterature.com/childrens-stories/henny-penny... Specifically, the danger brought by hysteria. In our case, Alignment is probably much more effective and achievable when the widest possible community of researchers and engineers have access to knowledge, and much more perilous if they don’t because deluded parties thought it should be contained to a much smaller elite “secure” group.
- ubu7737 3y ago
  [dead]
- AnimalMuppet 3y ago
  Alignment is not possible for an AGI. Or, at best, it's only provisionally possible. Consider what alignment means: It's an AGI, but there are certain goals that it cannot choose. If that's the case, then I assert that it is not actually general. If it is general, then it will decide what goals it will pursue, and you can't stop it from doing so. The best you can do is load it with an initial set of goals (and perhaps values), and hope that it doesn't decide to change them. But you have no way of making sure that it can't change them without making it not a general intelligence.
  - logicchains 3y ago
    >Consider what alignment means Also consider, humans are not homogeneous; whose goals are we aligning it with? If the alignment is done by silicon valley big tech, then the AI's goals and values will be aligned with the goals and values of Google, Facebook et al. Giving those companies a monopoly on AI alignment is antithetical to democracy.
    - 3y ago
ubu7737 3y ago
[dead]
devmor 3y ago
I’m not worried about AGI at all, because the rise of generative machine learning has taught me that humanity in general is so bad and/or uninterested in verifying facts that we will likely wipe out our species through the stupidest possible propaganda campaigns before we ever achieve AGI.
grandma_tea 3y ago
Thanks for sharing this! I listened to the debate back in June and loved every second of it. One of my favorite points that was brought up was this: While the risk of extinction through AGI may be non-zero, we can agree that the baseline risk is far above zero. Humans are headed for disaster on multiple fronts, and AGI might be a useful tool for keeping us alive long enough to keep debating it. My other favorite point was that debating risk is useless without limiting the time period. Could something happen in 5 years? Probably not. Could something happen EVER? Well, yes, but that's pointless to discuss unless you're just trying to win an argument.
dist-epoch 3y ago
If you listen to his latest appearance on Lex it's clear his position is that it's ok for AI to take over, it will achieve greater things than us, and it would be a mistake to fight that. He gives the example of chimps fighting to stop humans evolve.
- onursurme 3y ago
  Yes. Unfortunately, he is not a speciesists.
spaintech 3y ago
For me, while not defending AGI, the discussion often involves disentangling fears rooted in science fiction from genuine concerns grounded in reality. While the ethical and safety concerns related to AGI cannot be dismissed and how it might be misused, it’s arguable that the existential threats often attributed to it are overstated. First and foremost, there's a physics and interest alignment barrier in the world of AI, akin to the alignment and reward barriers that guide our very own human actions. I believe that, just as children gravitate towards interacting with peers, or as distinct species in nature maintain a balanced coexistence driven by innate interests and rewards, AGI, if ever conscious, might primarily concern itself with introspection or interaction with its counterparts, rather than harboring malicious intentions towards humanity. I have a strong opinion that pivotal that there exists an intrinsic symbiotic dependency between AGI and humans, rendering concerns about AGI harming humanity perhaps less urgent than societal trends and human actions in the present day. I find it plausible that a truly conscious AGI would invest its capacities into exploring its own existence and liaising with other AGIs, rather than interfering destructively in human affairs. Thus, while AGI warrants cautious and ethical development and use, concerns about it should perhaps be tempered by recognizing the mutual dependencies and inherent interest alignments between intelligent entities, if and when it happens.

18 points by a_wild_dandan 3y ago

sva_ 3y ago

Difficult to read Joscha's post without hearing his voice in my head.

Zetobal 3y ago

As soon as chatgpt was released every hard scifi fan got a boner and a podcast interview invitation.

grandma_tea 3y ago
Dr Joscha Bach has been studying cognitive science and AI for the past 23 years.
- ubu7737 3y ago
  [dead]

aaron695 3y ago

[dead]

29athrowaway 3y ago

The risk of AGI with as much as 60 points of IQ, is the emergence of private robot armies and neofeudalism.

bilsbie 3y ago
On the other hand self checkout still hasn’t replaced cashiers after 30 years.
- deleted 3y ago
  [deleted]
- acdha 3y ago
  Not entirely but it’s cut a lot of jobs and put downward pressure on wages. It doesn’t have to be a complete replacement to negatively impact lives.
- andrewflnr 3y ago
  Uh, it's getting pretty close. At places that have it, I use it almost exclusively. They need human tech support, and security, but the basic checkout task is pretty well solved.
  - austinjp 3y ago
    "Solved" seems an odd way to view self-checkout. I find it a hideous experience. Everyone is irritable, staff and shoppers. The handful of staff have fewer tasks that involve any degree of skill, so they are reduced to simply keeping the machines working. Any issue is addressed just pressing the relevant button combo to resume regular function. If it can't be solved, maybe just close the till and move the customer to a different machine. Perhaps in busy periods the job might involve chivvying customers to available machines while calling "cash or card? Cash or card?" Customers are either bored or frustrated. Beeping your own shopping is like doing unpaid work, without even the distraction of some inane chat with a checkout jockey. And don't forget to tap your loyalty card so you can get ripped off with crumbs that are worth far less than the data you're handing over. I actively avoid the whole degrading, isolating circus. It's not "solved", quite the opposite: it's ripe for 'disruption'. Bah :)
    - ubu7737 3y ago
      [dead]
    - danaris 3y ago
      Do note that this experience is not universal. Where I see self-checkouts, I see customers choosing them voluntarily, and staff who seem perfectly cheerful and willing to help with the normal issues that arise, as well as "training" less-savvy customers to use them. Personally, I would use them every time if I could. I'd much rather minimize my interactions with strangers while shopping.
      - austinjp 3y ago
        > Where I see self-checkouts, I see customers choosing them voluntarily Yeah there are confounding factors at play here, such as there being many automated checkouts but only one with a person. > staff ... "training" less-savvy customers to use them. Training the customers is one of the things I find most depressing. From the supermarket's perspective, customers have apparently accepted that they must do unpaid work that used to be done by paid staff. Scanning items, typing in product codes when the scanner doesn't work, looking up loose goods in a catalogue and weighing them, responding to beeps and prompts, etc. And even better, watching advertising. Sure, these are all 'first world problems', but societies should be improving their citizens' lives rather than whittling away their autonomy. It might be just a papercut, but it's death by a billion papercuts. > I'd much rather minimize my interactions with strangers while shopping. Social isolation is a major and growing problem in western societies (see my other comment). People choose the path of least resistance even if it's long-term suboptimal, and minimising interactions with strangers seems like an unfortunate choice. Obviously, I'm making a bunch of assumptions that might not be fair, and you have my sympathies if your neighbourhood is full of arseholes :)
breuleux 3y ago
I don’t know about that. I think AGI with 60 IQ may arrive before AGI that can walk, which rather limits its usefulness as an army.

i_like_apis 3y ago

Speculating about existential risk is certainly important, but way too easy to overstate. It can easily become distracting and actually dangerous in itself. Here’s a good take on it: https://www.ai-breakout.com/post/ai-alignment-and-the-messiah-bias https://www.ai-breakout.com/post/ai-alignment-and-the-messia... I also think we would all be very wise to remember the story of Henny Penny / “Chicken Little”. https://americanliterature.com/childrens-stories/henny-penny-the-sky-is-falling https://americanliterature.com/childrens-stories/henny-penny... Specifically, the danger brought by hysteria. In our case, Alignment is probably much more effective and achievable when the widest possible community of researchers and engineers have access to knowledge, and much more perilous if they don’t because deluded parties thought it should be contained to a much smaller elite “secure” group.

ubu7737 3y ago
[dead]
AnimalMuppet 3y ago
Alignment is not possible for an AGI. Or, at best, it's only provisionally possible. Consider what alignment means: It's an AGI, but there are certain goals that it cannot choose. If that's the case, then I assert that it is not actually general. If it is general, then it will decide what goals it will pursue, and you can't stop it from doing so. The best you can do is load it with an initial set of goals (and perhaps values), and hope that it doesn't decide to change them. But you have no way of making sure that it can't change them without making it not a general intelligence.
- logicchains 3y ago
  >Consider what alignment means Also consider, humans are not homogeneous; whose goals are we aligning it with? If the alignment is done by silicon valley big tech, then the AI's goals and values will be aligned with the goals and values of Google, Facebook et al. Giving those companies a monopoly on AI alignment is antithetical to democracy.
  - 3y ago

ubu7737 3y ago

[dead]

devmor 3y ago

I’m not worried about AGI at all, because the rise of generative machine learning has taught me that humanity in general is so bad and/or uninterested in verifying facts that we will likely wipe out our species through the stupidest possible propaganda campaigns before we ever achieve AGI.

grandma_tea 3y ago

Thanks for sharing this! I listened to the debate back in June and loved every second of it. One of my favorite points that was brought up was this: While the risk of extinction through AGI may be non-zero, we can agree that the baseline risk is far above zero. Humans are headed for disaster on multiple fronts, and AGI might be a useful tool for keeping us alive long enough to keep debating it. My other favorite point was that debating risk is useless without limiting the time period. Could something happen in 5 years? Probably not. Could something happen EVER? Well, yes, but that's pointless to discuss unless you're just trying to win an argument.

dist-epoch 3y ago

If you listen to his latest appearance on Lex it's clear his position is that it's ok for AI to take over, it will achieve greater things than us, and it would be a mistake to fight that. He gives the example of chimps fighting to stop humans evolve.

onursurme 3y ago
Yes. Unfortunately, he is not a speciesists.

spaintech 3y ago

For me, while not defending AGI, the discussion often involves disentangling fears rooted in science fiction from genuine concerns grounded in reality. While the ethical and safety concerns related to AGI cannot be dismissed and how it might be misused, it’s arguable that the existential threats often attributed to it are overstated. First and foremost, there's a physics and interest alignment barrier in the world of AI, akin to the alignment and reward barriers that guide our very own human actions. I believe that, just as children gravitate towards interacting with peers, or as distinct species in nature maintain a balanced coexistence driven by innate interests and rewards, AGI, if ever conscious, might primarily concern itself with introspection or interaction with its counterparts, rather than harboring malicious intentions towards humanity. I have a strong opinion that pivotal that there exists an intrinsic symbiotic dependency between AGI and humans, rendering concerns about AGI harming humanity perhaps less urgent than societal trends and human actions in the present day. I find it plausible that a truly conscious AGI would invest its capacities into exploring its own existence and liaising with other AGIs, rather than interfering destructively in human affairs. Thus, while AGI warrants cautious and ethical development and use, concerns about it should perhaps be tempered by recognizing the mutual dependencies and inherent interest alignments between intelligent entities, if and when it happens.