> Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, That may well be the ori

> Well, since the paperclip story is only trying to show that even very simple goals can lead to large impacts in the hands of an AGI, That may well be the original intention. But that's not how it's used in practice, is it? It's cited as an argument that AGI (in general) is unsafe. But it isn't an argument that AGI is unsafe! It's an argument that says that single-objective optimisation can/will violate some of the constraints we did not encode but really want it to respect. But who cares about that? It's a completely irreal thought adventure, and has no bearing on the actual problem or its possible solutions. Of course we can say: look, it's a scary thought! But only if we simultaneously admit that it has nothing whatsoever to do with the actual technology. The only thing it says anything about is a toy variation that noone in their right mind would ever consider building. > I take this to mean that you think one can bound the impacts if one chooses a complex enough set of goals. Ehh, precisely. I would actually go further and say that it is fairly simple to do so. If you can tell it to build paperclips, and expect it to understand that, you can also tell it not to damage or disrupt the ecosystem in the process, directly or indirectly. (Or, to strip away the last shreds of this idea: you can tell it to make no more than 100 million paper clips, and to rest on Sundays.) > Personally, I would read the actual academic making the argument rather than reposts of it by folks on the internet. Yudkowsky [1], who runs that site (which I don't frequent btw), seems to be pretty close to Bostrom. They've collaborated in the past. And if Bostrom's arguments can't be paraphrased (by his own friends, no less) without losing validity, that doesn't really seem like much of an endorsement either. I have nothing against these people. But please stop alluding to hidden treasure troves of arguments that cannot be reproduced and made apparent. I've asked enough times in this thread already if there is any highlight in there. I have a background in philosophy myself, and I know what to expect if I did pick up this book. And in that vein I really feel no need to do so. [1]: http://en.wikipedia.org/wiki/Eliezer_Yudkowsky http://en.wikipedia.org/wiki/Eliezer_Yudkowsky