5 ms·

This is an inaccurate and ignorant comment that does not engage with the arguments people on LessWrong are making. If you have something smart to say about why

This is an inaccurate and ignorant comment that does not engage with the arguments people on LessWrong are making. If you have something smart to say about why these people are wrong in their beliefs, please do. Name calling a group of people a "cult" or "apocalyptic religion" is not an argument that contributes anything interesting to the conversation. If anyone is actually curious about what these people are saying, here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... If anyone has a coherent counter-argument to any of that, I'd really love to hear it.

antonvs 4y ago
A good intro for why they’re wrong in their beliefs is https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
- cwmoore 4y ago
  That highlights the actual problem: > It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible.
version_five 4y ago
here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all Not familiar with them in the least, but the fact that they're a group that's assembled such a list basically proves they're something of a doomsday cult, no?
- ctoth 4y ago
  Man, wait until you learn about the climate change movement.
cwmoore 4y ago
That link is not at all a list of "why AI is extremely dangerous and extremely likely to kill us all". I skimmed it (there's a lot of writing assuming grave danger) but found only the following example of AI lethality, and it appears fantastical: > ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
- eternalban 4y ago
  https://news.ycombinator.com/item?id=35402539 https://news.ycombinator.com/item?id=35402539 mind viruses.
  - cwmoore 4y ago
    > The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors. There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?
- uxp100 4y ago
  It’s interesting how the Eric Drexler influence clearly colors a lot of apocalyptic AI stuff, in fiction, and on Less Wrong. First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.
- cubefox 4y ago
  Here is a much more realistic example: https://twitter.com/RokoMijic/status/1641806556038676481 https://twitter.com/RokoMijic/status/1641806556038676481 In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.
joshuahedlund 4y ago
> My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.
- a_shovel 4y ago
  (You quoted the same paragraph I did, so I'll just delete my comment and move my response to this quotation here.) Exactly. Why am I supposed to take this seriously? Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.
- cubefox 4y ago
  You are demanding "believable science" from a chimpanzee of how humans would overpower them. Do you not recognize the enormous difference intelligence makes? And there does not seem to be a hard constraint of how much more intelligent than a human an AI could become. Their brains are not tightly restricted by size and energy. The difference could be much greater than between chimpanzee and human.
23B1 4y ago
The easiest counterargument is that humans are provably, systematically, and consistently awful at predicting the future.
deleted 4y ago
[deleted]
evilduck 4y ago
Find and replace "God" for "AGI" and a lot of that link's reasoning will look awfully familiar to atheist circles.

antonvs 4y ago

A good intro for why they’re wrong in their beliefs is https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm

cwmoore 4y ago
That highlights the actual problem: > It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible.

version_five 4y ago

here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all Not familiar with them in the least, but the fact that they're a group that's assembled such a list basically proves they're something of a doomsday cult, no?

ctoth 4y ago
Man, wait until you learn about the climate change movement.

cwmoore 4y ago

That link is not at all a list of "why AI is extremely dangerous and extremely likely to kill us all". I skimmed it (there's a lot of writing assuming grave danger) but found only the following example of AI lethality, and it appears fantastical: > ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.

eternalban 4y ago
https://news.ycombinator.com/item?id=35402539 https://news.ycombinator.com/item?id=35402539 mind viruses.
- cwmoore 4y ago
  > The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors. There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?
uxp100 4y ago
It’s interesting how the Eric Drexler influence clearly colors a lot of apocalyptic AI stuff, in fiction, and on Less Wrong. First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.
cubefox 4y ago
Here is a much more realistic example: https://twitter.com/RokoMijic/status/1641806556038676481 https://twitter.com/RokoMijic/status/1641806556038676481 In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.

joshuahedlund 4y ago

> My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.

a_shovel 4y ago
(You quoted the same paragraph I did, so I'll just delete my comment and move my response to this quotation here.) Exactly. Why am I supposed to take this seriously? Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.
cubefox 4y ago
You are demanding "believable science" from a chimpanzee of how humans would overpower them. Do you not recognize the enormous difference intelligence makes? And there does not seem to be a hard constraint of how much more intelligent than a human an AI could become. Their brains are not tightly restricted by size and energy. The difference could be much greater than between chimpanzee and human.

23B1 4y ago

The easiest counterargument is that humans are provably, systematically, and consistently awful at predicting the future.

deleted 4y ago

[deleted]

evilduck 4y ago

Find and replace "God" for "AGI" and a lot of that link's reasoning will look awfully familiar to atheist circles.