The point of the thought experiment is that there's a difference between intelligence and goals. There are other thought experiments (and just general study of

The point of the thought experiment is that there's a difference between intelligence and goals. There are other thought experiments (and just general study of human cognition) whose point is that accurately capturing human goals and values is hard, possibly harder than making a general intelligence with X goals in the first place. (See http://wiki.lesswrong.com/wiki/Complexity_of_value http://wiki.lesswrong.com/wiki/Complexity_of_value) So organizations like MIRI exist (http://intelligence.org http://intelligence.org) to try and solve this problem sooner rather than later, because once a more-than-human-intelligent agent is running, despite the controls, if its values aren't precise enough and if they aren't stable enough on improvement, there is immense possibility for failure. It's also sort of questionable to talk about effective controls of something that's smarter and faster than you are. (See the AI Box Experiments, it's suggestive that you don't even need anything beyond human intelligence to subvert controls you don't like. http://www.yudkowsky.net/singularity/aibox http://www.yudkowsky.net/singularity/aibox) These solar system tiling examples are just a dramatic case for something terrible that could happen given a generally intelligent machine with non-human-friendly goals, or even friendly-seeming goals (like "make nanomachines that remove cancerous cells") that are improperly specified to cover corner cases, but if you spend time analyzing more mundane ways things could go slightly wrong to terribly wrong given an honest but flawed attempt at making sure they go right, carry your analyses years into the future after the intelligent software is started where things continue going right but then go wrong, you might come to agree that the most likely outcome given present knowledge and research direction will be bad for humanity.