4 ms·

Agreed point 2 seems to be the crux. An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more re

Agreed point 2 seems to be the crux. An important concept behind this is Omohundro's Basic Drives. Any maximizing agent with a goal will try to acquire more resources, resist being shut off, resist having its goal changed, create copies of itself, and improve its algorithm. If it is possible to maximize its goal in a way that will not guarantee humanity's flourishing, then we all die, guaranteed. If you want something more spelled out, I'll refer to Tim Urban's explanation. It's quite long, but is about as detailed as you can ask for. https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol...

aeternum 4y ago
Yes point 2 is the weak one. Aren't humans/biological life a counterexample? Simple bacteria are clearly maximizing agents, and cyanobacteria did infact almost destroy all life on earth by filling the entire planet with toxic oxygen. We've known how to improve ourselves via selective breeding yet vanishingly few humans are proponents. We and other intelligent animals have a wide variety of goals, and even share food across species. The evidence just doesn't seem to support this concept of Basic Drives. If anything, the evidence (and common sense) seems to suggest that the more intelligent the organism, the more easily and more often it ignores its basic drives.
- gonehome 4y ago
  Natural selection and biology have different constraints that limit things (and even then often selection can cause extinction) Intellectual capability and alignment are orthogonal - the former doesn’t get you the latter for free. Most people new to this issue don’t intuitively grasp this at first.
  - aeternum 4y ago
    What evidence do you have that intellectual ability and alignment are orthogonal? Remember that ML models are also subject to selection. If they're not useful/beneficial to us, we stop running them.

aeternum 4y ago

Yes point 2 is the weak one. Aren't humans/biological life a counterexample? Simple bacteria are clearly maximizing agents, and cyanobacteria did infact almost destroy all life on earth by filling the entire planet with toxic oxygen. We've known how to improve ourselves via selective breeding yet vanishingly few humans are proponents. We and other intelligent animals have a wide variety of goals, and even share food across species. The evidence just doesn't seem to support this concept of Basic Drives. If anything, the evidence (and common sense) seems to suggest that the more intelligent the organism, the more easily and more often it ignores its basic drives.

gonehome 4y ago
Natural selection and biology have different constraints that limit things (and even then often selection can cause extinction) Intellectual capability and alignment are orthogonal - the former doesn’t get you the latter for free. Most people new to this issue don’t intuitively grasp this at first.
- aeternum 4y ago
  What evidence do you have that intellectual ability and alignment are orthogonal? Remember that ML models are also subject to selection. If they're not useful/beneficial to us, we stop running them.