Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets

Imagine someone develops a really big A* algorithm with many clever optimizations, gives it a goal of choosing actions that lead to maximum paperclips, and sets it loose upon the world. Let's call it Clippy. I hope it's clear that Clippy will read the Bostrom book and use the information to maximize paperclips, rather than do something it wasn't programmed to do like "help humanity". I'm not interested in arguing whether Clippy is truly "intelligent", let's leave that to the dictionary makers. But there's no law of nature saying you can't make Clippy, or that it won't be dangerous once made. All the evidence we have right now says that it's possible. The big problem right now is that many organizations are working to develop the moral equivalent of Clippy, they are making visible progress, they are in an arms race with each other so we shouldn't hope for restraint, and we have absolutely no idea how to make it safe.