4 ms·

That link is not at all a list of "why AI is extremely dangerous and extremely likely to kill us all". I skimmed it (there's a lot of writing assuming grave dan

That link is not at all a list of "why AI is extremely dangerous and extremely likely to kill us all". I skimmed it (there's a lot of writing assuming grave danger) but found only the following example of AI lethality, and it appears fantastical: > ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.

eternalban 4y ago
https://news.ycombinator.com/item?id=35402539 https://news.ycombinator.com/item?id=35402539 mind viruses.
- cwmoore 4y ago
  > The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors. There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?
uxp100 4y ago
It’s interesting how the Eric Drexler influence clearly colors a lot of apocalyptic AI stuff, in fiction, and on Less Wrong. First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.
cubefox 4y ago
Here is a much more realistic example: https://twitter.com/RokoMijic/status/1641806556038676481 https://twitter.com/RokoMijic/status/1641806556038676481 In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.
- cwmoore 4y ago
  I'm just not seeing any real examples, exotic or otherwise, of how to recognize the risks involved in superhuman AI. Smarter than us does not imply any particular will or agency. Of course the will and agency of the people employing the technology will reveal its immediate impact, likely toward short term profit, and alignment as a control plane employed toward the same. Chimpanzees have limited control over humans, humans have limited control over their own technology. But of course chimpanzees can recognize the risk of armed people. The point is not to debunk fanciful end games. Better to prepare for likely effects on the human network. I was promised a list.
  - cubefox 4y ago
    "Recognize the risk" what does that mean? That advanced AI would could overpower us? That it is likely that it would want to overpower us?
    - cwmoore 4y ago
      Your link suggests the relative likelihood of AI destroying all carbon based life or saving the world from "various stupidities". If it is somewhere in between in its influence, and changes anything, isn't it still reasonable to call it a risk? OP promised a list of scenarios by which AGI leads to total devastation, with strong language for GP. For clarity, I don't think that it is very likely that an unimaginable silicon takeover is the most probable outcome. But like with the gun or the printing press, there will be winners and losers, and society will shift.

eternalban 4y ago

https://news.ycombinator.com/item?id=35402539 https://news.ycombinator.com/item?id=35402539 mind viruses.

cwmoore 4y ago
> The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors. There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?

uxp100 4y ago

It’s interesting how the Eric Drexler influence clearly colors a lot of apocalyptic AI stuff, in fiction, and on Less Wrong. First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.

cubefox 4y ago

Here is a much more realistic example: https://twitter.com/RokoMijic/status/1641806556038676481 https://twitter.com/RokoMijic/status/1641806556038676481 In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.

cwmoore 4y ago
I'm just not seeing any real examples, exotic or otherwise, of how to recognize the risks involved in superhuman AI. Smarter than us does not imply any particular will or agency. Of course the will and agency of the people employing the technology will reveal its immediate impact, likely toward short term profit, and alignment as a control plane employed toward the same. Chimpanzees have limited control over humans, humans have limited control over their own technology. But of course chimpanzees can recognize the risk of armed people. The point is not to debunk fanciful end games. Better to prepare for likely effects on the human network. I was promised a list.
- cubefox 4y ago
  "Recognize the risk" what does that mean? That advanced AI would could overpower us? That it is likely that it would want to overpower us?
  - cwmoore 4y ago
    Your link suggests the relative likelihood of AI destroying all carbon based life or saving the world from "various stupidities". If it is somewhere in between in its influence, and changes anything, isn't it still reasonable to call it a risk? OP promised a list of scenarios by which AGI leads to total devastation, with strong language for GP. For clarity, I don't think that it is very likely that an unimaginable silicon takeover is the most probable outcome. But like with the gun or the printing press, there will be winners and losers, and society will shift.