3 ms·
Just because something is AGI probably needn't imply that it is sapient. It might have no internal experience, no feeling of desire, only a utility function whi
by drdeca 5y ago
Just because something is AGI probably needn't imply that it is sapient. It might have no internal experience, no feeling of desire, only a utility function which it optimizes.
- krageon 5y agoThis reasoning is why it will need rights. If it doesn't, people thinking these thoughts (which are impossible to answer, like they are for every other potential sapience. Not just AGI) will abuse that reasoning to create yet another creative modern form of slavery.
- drdeca 5y agoI do agree that it probably makes sense to err on the side of being too quick to estimate that something is likely sapient than to err on the side of being too slow to do so. However, that doesn't make it impossible for a plan-producer to be both superhuman and non-sapient. Nor do I think the default is for an AGI to be sapient. (Though, possibly the default could be such that it isn't but in a way in which we couldn't be sufficiently confident that it isn't, and therefore be obligated to behave as if it is sapient) But, in the FOOM scenario, I would expect that "whether or not to treat it as if it has rights" would be the least of our worries. (the greatest of our worries would be, roughly, "whether it kills everyone".)