3 ms·

Yeah, but the paper-clip maximizer theory is pretty awfully flawed. It rests on the idea that a generally human-level of intelligence necessarily leads to a sup

Yeah, but the paper-clip maximizer theory is pretty awfully flawed. It rests on the idea that a generally human-level of intelligence necessarily leads to a super-human explosion of intelligence. This is clearly not the case.

ikeboy 11y ago
>It rests on the idea that a generally human-level of intelligence necessarily leads to a super-human explosion of intelligence. Why must it be necessary? If there's a large chance, that's still a problem. Besides, the claim is generally that a human-created intelligence is unlikely to be in the same range as humans (because there's no reason to think that the human range on the spectrum is unique), so if it's not dumber, then it's most likely going to significantly smarter. There's also the point that if you have a human-level AI, simply throwing more computing power at it makes it strictly faster than humans. See http://intelligenceexplosion.com/en/2011/plenty-of-room-above-us/ http://intelligenceexplosion.com/en/2011/plenty-of-room-abov... and http://www.nickbostrom.com/superintelligence.html http://www.nickbostrom.com/superintelligence.html for some examples of these arguments.
Strilanc 11y ago
How does the existence of a paperclip maximizer rest on the assumption of recursive self-improvement? Making nanotech weapons to stop humans from shutting you down and thereby reducing the expected number of paperclips doesn't require exponentially increasing amounts of intelligence. It only requires a fixed amount of intelligence. Also you should justify "this is clearly [1] not the case". You're probably taking the "necessarily" phrasing really literally, instead of as "with non-negligible risk", and thinking of corporations or evolution as "superhuman"? On the other hand, evolution and corporations respectively did and plan to produce smarter entities so that's not so clear cut. 1: https://medium.com/science-and-technology/how-to-spot-a-weak-argument-83dacb1fe14c https://medium.com/science-and-technology/how-to-spot-a-weak...

ikeboy 11y ago

>It rests on the idea that a generally human-level of intelligence necessarily leads to a super-human explosion of intelligence. Why must it be necessary? If there's a large chance, that's still a problem. Besides, the claim is generally that a human-created intelligence is unlikely to be in the same range as humans (because there's no reason to think that the human range on the spectrum is unique), so if it's not dumber, then it's most likely going to significantly smarter. There's also the point that if you have a human-level AI, simply throwing more computing power at it makes it strictly faster than humans. See http://intelligenceexplosion.com/en/2011/plenty-of-room-above-us/ http://intelligenceexplosion.com/en/2011/plenty-of-room-abov... and http://www.nickbostrom.com/superintelligence.html http://www.nickbostrom.com/superintelligence.html for some examples of these arguments.

Strilanc 11y ago

How does the existence of a paperclip maximizer rest on the assumption of recursive self-improvement? Making nanotech weapons to stop humans from shutting you down and thereby reducing the expected number of paperclips doesn't require exponentially increasing amounts of intelligence. It only requires a fixed amount of intelligence. Also you should justify "this is clearly [1] not the case". You're probably taking the "necessarily" phrasing really literally, instead of as "with non-negligible risk", and thinking of corporations or evolution as "superhuman"? On the other hand, evolution and corporations respectively did and plan to produce smarter entities so that's not so clear cut. 1: https://medium.com/science-and-technology/how-to-spot-a-weak-argument-83dacb1fe14c https://medium.com/science-and-technology/how-to-spot-a-weak...