3 ms·

Interestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost

by taneq 5y ago

Interestingly I was just today explaining the paperclip optimizer scenario to a friend who asked about the dangers of AI, including the fact that there's almost no general optimization task that doesn't (with a sufficiently long lookahead) involve taking over the world as an intermediate step. (Obviously closed, specific tasks like "land this particular rocket safely within 15 minutes" don't always lead to this, but open ended ones like "manufacture mcguffins" or "bring about world peace" sure seem to.)

lancengym 5y ago
Perhaps all AI eventually figure out that humans are the REAL problems because we don't optimize, we lust and hoard and are envious and greedy - the very antithesis of resource optimization! Lol.
- taneq 5y ago
  We're just optimizing (generally quite well, I might add) for genetic survival.
  - NaturalPhallacy 5y ago
    Ian Banks did a really amazing exposition of this where the Culture was rallying to stamp out reproducing nanites and they had to be stopped because if not they'd literally turn the whole universe into copies of themselves. One of the human characters mused that isn't that what all life is trying to do? I think it was in the Hydrogen Sonata, but I'm not sure.
    - taneq 5y ago
      Yes! I often find myself thinking of organisms as 'hegemonising swarms.'
OscarCunningham 5y ago
> "land this particular rocket safely within 15 minutes" This one becomes especially dangerous after the 15 minutes have passed and it begins to concentrate all its attention on the paranoid scenarios where its timekeeping is wrong and 15 minutes haven't actually passed.
- taneq 5y ago
  Ooh true, that could generate some interesting scenarios. "No, it's the GPS satellite clocks that are wrong, I must destroy them before they corrupt the world and cause another rocket to land at the wrong time!"
XorNot 5y ago
Always a good time to post Jipi and the Paranoid Chip: https://vanemden.com/books/neals/jipi.html https://vanemden.com/books/neals/jipi.html Which pretty much tackles these issues head on.
deleted 5y ago
[deleted]

by taneq 5y ago

lancengym 5y ago

Perhaps all AI eventually figure out that humans are the REAL problems because we don't optimize, we lust and hoard and are envious and greedy - the very antithesis of resource optimization! Lol.

taneq 5y ago
We're just optimizing (generally quite well, I might add) for genetic survival.
- NaturalPhallacy 5y ago
  Ian Banks did a really amazing exposition of this where the Culture was rallying to stamp out reproducing nanites and they had to be stopped because if not they'd literally turn the whole universe into copies of themselves. One of the human characters mused that isn't that what all life is trying to do? I think it was in the Hydrogen Sonata, but I'm not sure.
  - taneq 5y ago
    Yes! I often find myself thinking of organisms as 'hegemonising swarms.'

OscarCunningham 5y ago

> "land this particular rocket safely within 15 minutes" This one becomes especially dangerous after the 15 minutes have passed and it begins to concentrate all its attention on the paranoid scenarios where its timekeeping is wrong and 15 minutes haven't actually passed.

taneq 5y ago
Ooh true, that could generate some interesting scenarios. "No, it's the GPS satellite clocks that are wrong, I must destroy them before they corrupt the world and cause another rocket to land at the wrong time!"

XorNot 5y ago

Always a good time to post Jipi and the Paranoid Chip: https://vanemden.com/books/neals/jipi.html https://vanemden.com/books/neals/jipi.html Which pretty much tackles these issues head on.

deleted 5y ago

[deleted]