In an ideal world, I think this would work out. The problem is, I can't trust anyone to create an AI that's truly unbiased. > the things we find desirable in

In an ideal world, I think this would work out. The problem is, I can't trust anyone to create an AI that's truly unbiased. > the things we find desirable in our legal decisions. Who is "we"? Everyone has different motives, so having a human create an optimization function won't create a truly unbiased machine. I think any good AI will have to create its own "opinions" to be of any worth.