4 ms·

"Purely theoretical" is too weak. "Physically impossible" is better. AI Safety guys really like using the physically impossible to advance their arguments. It'

"Purely theoretical" is too weak. "Physically impossible" is better. AI Safety guys really like using the physically impossible to advance their arguments. It's bizarre! Pure fantasy isn't something worth reacting to.

jandrese 6y ago
It's the classic case of seeing a curve pointing upwards and thinking it will continue doing that forever, even though the universe is like 0 for nearly-uncountable in cases where that has held true indefinitely. Every growth curve is an S curve. The AI Singularity is like the Grey Goo scenario. An oversimplified model projected out so far in the future that its flaws become apparent.
- ben_w 6y ago
  Depends which version of “the singularity” is being discussed. IIRC, the original (almost certainly false) idea was a sufficiently powerful AI can start a sequence of ever power powerful AI with decreasing time between each step — reaching infinite power in finite time. I don’t need that version of the singularity to be worried. I think in terms of “the event horizon” rather than “the singularity”: all new tech changes the world, when the rate of change exceeds our capacity to keep up with the consequences, stuff will go wrong on a large scale for lots of people. As for grey goo? Self replicating nanomachines is just biology. It gets everywhere, and even single-celled forms can kill you by eating your flesh or suborning your cells, but it’s mostly no big deal because you evolved to deal with that threat.
  - jandrese 6y ago
    The Grey Goo scenario is that it starts eating the planet until the only thing left is a sea of nanomachines. However, thermodynamics says it becomes increasingly more difficult to distribute power (food) to a growing population and it hits a natural limit to growth, just like bacteria.
postalrat 6y ago
"Circle" and "exponential growth" are also physically impossible yet useful ideas.

jandrese 6y ago

It's the classic case of seeing a curve pointing upwards and thinking it will continue doing that forever, even though the universe is like 0 for nearly-uncountable in cases where that has held true indefinitely. Every growth curve is an S curve. The AI Singularity is like the Grey Goo scenario. An oversimplified model projected out so far in the future that its flaws become apparent.

ben_w 6y ago
Depends which version of “the singularity” is being discussed. IIRC, the original (almost certainly false) idea was a sufficiently powerful AI can start a sequence of ever power powerful AI with decreasing time between each step — reaching infinite power in finite time. I don’t need that version of the singularity to be worried. I think in terms of “the event horizon” rather than “the singularity”: all new tech changes the world, when the rate of change exceeds our capacity to keep up with the consequences, stuff will go wrong on a large scale for lots of people. As for grey goo? Self replicating nanomachines is just biology. It gets everywhere, and even single-celled forms can kill you by eating your flesh or suborning your cells, but it’s mostly no big deal because you evolved to deal with that threat.
- jandrese 6y ago
  The Grey Goo scenario is that it starts eating the planet until the only thing left is a sea of nanomachines. However, thermodynamics says it becomes increasingly more difficult to distribute power (food) to a growing population and it hits a natural limit to growth, just like bacteria.

postalrat 6y ago

"Circle" and "exponential growth" are also physically impossible yet useful ideas.