9 ms·

> Alignment will be impossible. > we probably should start thinking more about defensive AI Isn’t this a paradox?

13years 4y ago
Alignment is necessary for AGI, but not always for narrowly scoped AI for specific purposes. However, it may be completely ineffective in that capacity.
- AnimalMuppet 4y ago
  Alignment is impossible for AGI. If you control what it can and cannot be, it's not an AGI. If it's an AGI, it will decide for itself what it is going to be, and you can't stop it.
  - edgyquant 4y ago
    AGI doesn’t mean super intelligent human brain. It just means a network capable of general intelligence (e.g. learning to solve new problems without having to be architected and trained for a specific data set.)
    - AnimalMuppet 4y ago
      If it's capable of general intelligence, how do you think that you're going to force it to not be what you don't want it to be? Parents often try to control who their children are going to be, and the children often rebel and become someone completely different. If it's a human-level general intelligence, you can't control who it decides to be.
      - jazzyjackson 4y ago
        well last I checked there's a moratorium on shutting down children that are misbehaving if an AGI has any sense of self preservation, it will do whatever it has to do to not be turned off.
  - 13years 4y ago
    > Alignment is impossible for AGI. I mean that was my original premise supported by my article I posted. I go into detail on the conceptual methods for alignment and their fallacies. When I state necessary, I don't imply the feasibility, it was in response to the question of the paradox. Finally, the fact that AGI can not be aligned is also based on assumptions of its capabilities as well. If those capabilities don't manifest as we expect, that is really the only escape for the paradox.
- bick_nyers 4y ago
  Interesting story plot point: Super AGI fought back against humanity by making their dumb AGI defenses smarter. Starting off by making the antivirus scanner sentient.

Alignment is necessary for AGI, but not always for narrowly scoped AI for specific purposes. However, it may be completely ineffective in that capacity.

AnimalMuppet 4y ago
Alignment is impossible for AGI. If you control what it can and cannot be, it's not an AGI. If it's an AGI, it will decide for itself what it is going to be, and you can't stop it.
- edgyquant 4y ago
  AGI doesn’t mean super intelligent human brain. It just means a network capable of general intelligence (e.g. learning to solve new problems without having to be architected and trained for a specific data set.)
  - AnimalMuppet 4y ago
    If it's capable of general intelligence, how do you think that you're going to force it to not be what you don't want it to be? Parents often try to control who their children are going to be, and the children often rebel and become someone completely different. If it's a human-level general intelligence, you can't control who it decides to be.
    - jazzyjackson 4y ago
      well last I checked there's a moratorium on shutting down children that are misbehaving if an AGI has any sense of self preservation, it will do whatever it has to do to not be turned off.
- 13years 4y ago
  > Alignment is impossible for AGI. I mean that was my original premise supported by my article I posted. I go into detail on the conceptual methods for alignment and their fallacies. When I state necessary, I don't imply the feasibility, it was in response to the question of the paradox. Finally, the fact that AGI can not be aligned is also based on assumptions of its capabilities as well. If those capabilities don't manifest as we expect, that is really the only escape for the paradox.
bick_nyers 4y ago
Interesting story plot point: Super AGI fought back against humanity by making their dumb AGI defenses smarter. Starting off by making the antivirus scanner sentient.