2 ms·

> My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Interne

> My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer. Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.

a_shovel 4y ago
(You quoted the same paragraph I did, so I'll just delete my comment and move my response to this quotation here.) Exactly. Why am I supposed to take this seriously? Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.
cubefox 4y ago
You are demanding "believable science" from a chimpanzee of how humans would overpower them. Do you not recognize the enormous difference intelligence makes? And there does not seem to be a hard constraint of how much more intelligent than a human an AI could become. Their brains are not tightly restricted by size and energy. The difference could be much greater than between chimpanzee and human.
- joshuahedlund 4y ago
  But that’s just an unfalsifiable argument that could literally argue for anything. “I’m worried AIs will figure out how to go back in time and kill our last universal common ancestor.” “Explain how that is even believable.” “AI will be too smart, I can’t.” That’s not an argument; the burden of proof is to at least explain why the proposed doom scenario is not incompatible with the laws of physics. > Their brains are not tightly restricted by size and energy. I’m not sure that’s true. Computer cycles cost real money. And the scaling required to get us to GTP-4 makes me think, assuming super intelligent AGI is even possible, it’s going to have to run up a pretty noticeable AWS bill…
  - cubefox 4y ago
    For time travel there is are physical arguments why it may be impossible in principle , but there are obviously no such arguments for technology which overpowers humans. I don't think any specific example is needed to show that humans can be overpowered by an superior intellect. Just imagine highly advanced aliens visit Earth tomorrow. Yes, compute costs money, but for the human brain energy was way more restricted, since an substantial increase in energy demand would lengthen the time needed for finding food.
- revelio 4y ago
  Chimpanzees are extremely dangerous when angry and can easily overpower a human. It took a long time for humanity to get to the point of being able to fight off a troop of wild chimps and a lot of the effort required wasn't simply being smarter, it involved a lot of time consuming building and risk taking.

a_shovel 4y ago

(You quoted the same paragraph I did, so I'll just delete my comment and move my response to this quotation here.) Exactly. Why am I supposed to take this seriously? Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.

cubefox 4y ago

You are demanding "believable science" from a chimpanzee of how humans would overpower them. Do you not recognize the enormous difference intelligence makes? And there does not seem to be a hard constraint of how much more intelligent than a human an AI could become. Their brains are not tightly restricted by size and energy. The difference could be much greater than between chimpanzee and human.

joshuahedlund 4y ago
But that’s just an unfalsifiable argument that could literally argue for anything. “I’m worried AIs will figure out how to go back in time and kill our last universal common ancestor.” “Explain how that is even believable.” “AI will be too smart, I can’t.” That’s not an argument; the burden of proof is to at least explain why the proposed doom scenario is not incompatible with the laws of physics. > Their brains are not tightly restricted by size and energy. I’m not sure that’s true. Computer cycles cost real money. And the scaling required to get us to GTP-4 makes me think, assuming super intelligent AGI is even possible, it’s going to have to run up a pretty noticeable AWS bill…
- cubefox 4y ago
  For time travel there is are physical arguments why it may be impossible in principle , but there are obviously no such arguments for technology which overpowers humans. I don't think any specific example is needed to show that humans can be overpowered by an superior intellect. Just imagine highly advanced aliens visit Earth tomorrow. Yes, compute costs money, but for the human brain energy was way more restricted, since an substantial increase in energy demand would lengthen the time needed for finding food.
revelio 4y ago
Chimpanzees are extremely dangerous when angry and can easily overpower a human. It took a long time for humanity to get to the point of being able to fight off a troop of wild chimps and a lot of the effort required wasn't simply being smarter, it involved a lot of time consuming building and risk taking.