Because we're talking about ethics here, not just optimization. Ethics involves finding out what's wrong with reality and working to fix it, no matter how hard

Because we're talking about ethics here, not just optimization. Ethics involves finding out what's wrong with reality and working to fix it, no matter how hard it may seem. We need AI that doesn't simply accept reality "as is", but can also help us fix it. Otherwise AI will be little more than hyper-optimized reflections of our own failures and prejudices. We've had plenty of that already, we don't really need more of it. "The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man." - George Bernard Shaw