3 ms·

I think you are homing in on a key philosophical difference between me and LessWrongsters. Don't really have time to get into it now, except to say that it is k

by mtraven 13y ago

I think you are homing in on a key philosophical difference between me and LessWrongsters. Don't really have time to get into it now, except to say that it is kind of arrogant to think you can design or think about superintelligences without reference to the best existing intelligent systems we have. Especially if you want to keep them goal-compatible. The pathologies of such systems are especially instructive.

gwern 13y ago
> it is kind of arrogant to think you can design or think about superintelligences without reference to the best existing intelligent systems we have. I don't think it's any more arrogant than, say, Djikstra pointing out that submarines move in a completely different manner than a human swimmer. How arrogant were the Wright brothers in looking at birds and deciding to try to achieve some of the same goals by a completely different mechanism? If computers thought the same way humans did, if they came with builtin moral sentiments, the product of a very unusual evolutionary history and social structure, if they had access to no novel capabilities, then humans and forms of cooperation developed over the last few eyeblinks (centuries) might be relevant. But then, no one would care about the issue in the first place...