3 ms·

What about "most arbitrary goals are incompatible with human existence" requires super-human intelligence? A human who wanted to "build as many paperclips as p

What about "most arbitrary goals are incompatible with human existence" requires super-human intelligence? A human who wanted to "build as many paperclips as possible" could cause a great deal of destruction today. A human who wanted to accumulate as much wealth as possible could, too. EDIT: maybe a better way of articulating my complaints about this famous thought experiment is that it's supposed to be making a point about superintelligence but it's talking about a goal that has sub-human-intelligence sophistication.

circuit10 3y ago
> What about "most arbitrary goals are incompatible with human existence" requires super-human intelligence? The "taken to the absolute extreme" part. > A human who wanted to "build as many paperclips as possible" could cause a great deal of destruction today. Maybe, but a) no one really wants that (at least not as their only desire above all else) and b) we aren't superintelligent so it's hard to gain enough control and power and plan well enough to do it that well > talking about a goal that has sub-human-intelligence sophistication There is no reason a simple goal can't be followed in an intelligent way or vice versa. This is called the "orthogonality thesis". There's a good video about it here: https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo