4 ms·

Yes it is my first exposure to this thought experiment and I do not have trouble understanding the thought experiment. I am just going ahead and applying some c

by rsp1984 11y ago

Yes it is my first exposure to this thought experiment and I do not have trouble understanding the thought experiment. I am just going ahead and applying some common sense and logic, to conclude that the argument is pretty much entirely theoretical. Yes, optimizing only for maximum number of paper clips could potentially have some bad side effects, I get that. If that's the point of the thought experiment, fine. However that's not how the author of the blog post put it: He expressed concern that this could happen in real life, in the future. And I don't think it could. Why? Because in real life we'd not invent a super intelligent machine and then feed it with some objective function to maximize and then let it do its thing, watch it go out of control and destroy earth. In real life we'd make sure we're in control over that machine. In real life we'd make sure we put very clear and enforceable mechanisms into that machine to stop it from doing anything harmful in the first place while it is carrying out steps to reach its objective. In real life should we still see it doing something funny we pull the plug. End of story. In addition: Implementing above mentioned mechanisms is probably the easier part of the whole exercise. The hard bit is inventing a machine that can build a paper clip factory. If we can invent such a machine by then we certainly have also invented mechanisms to control that machine and only have it do "good" stuff.

Jach 11y ago
The point of the thought experiment is that there's a difference between intelligence and goals. There are other thought experiments (and just general study of human cognition) whose point is that accurately capturing human goals and values is hard, possibly harder than making a general intelligence with X goals in the first place. (See http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value) So organizations like MIRI exist (http://intelligence.org http://intelligence.org) to try and solve this problem sooner rather than later, because once a more-than-human-intelligent agent is running, despite the controls, if its values aren't precise enough and if they aren't stable enough on improvement, there is immense possibility for failure. It's also sort of questionable to talk about effective controls of something that's smarter and faster than you are. (See the AI Box Experiments, it's suggestive that you don't even need anything beyond human intelligence to subvert controls you don't like. http://www.yudkowsky.net/singularity/aibox http://www.yudkowsky.net/singularity/aibox) These solar system tiling examples are just a dramatic case for something terrible that could happen given a generally intelligent machine with non-human-friendly goals, or even friendly-seeming goals (like "make nanomachines that remove cancerous cells") that are improperly specified to cover corner cases, but if you spend time analyzing more mundane ways things could go slightly wrong to terribly wrong given an honest but flawed attempt at making sure they go right, carry your analyses years into the future after the intelligent software is started where things continue going right but then go wrong, you might come to agree that the most likely outcome given present knowledge and research direction will be bad for humanity.

by rsp1984 11y ago

Jach 11y ago

The point of the thought experiment is that there's a difference between intelligence and goals. There are other thought experiments (and just general study of human cognition) whose point is that accurately capturing human goals and values is hard, possibly harder than making a general intelligence with X goals in the first place. (See http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value) So organizations like MIRI exist (http://intelligence.org http://intelligence.org) to try and solve this problem sooner rather than later, because once a more-than-human-intelligent agent is running, despite the controls, if its values aren't precise enough and if they aren't stable enough on improvement, there is immense possibility for failure. It's also sort of questionable to talk about effective controls of something that's smarter and faster than you are. (See the AI Box Experiments, it's suggestive that you don't even need anything beyond human intelligence to subvert controls you don't like. http://www.yudkowsky.net/singularity/aibox http://www.yudkowsky.net/singularity/aibox) These solar system tiling examples are just a dramatic case for something terrible that could happen given a generally intelligent machine with non-human-friendly goals, or even friendly-seeming goals (like "make nanomachines that remove cancerous cells") that are improperly specified to cover corner cases, but if you spend time analyzing more mundane ways things could go slightly wrong to terribly wrong given an honest but flawed attempt at making sure they go right, carry your analyses years into the future after the intelligent software is started where things continue going right but then go wrong, you might come to agree that the most likely outcome given present knowledge and research direction will be bad for humanity.