The worrying scenario is just any powerful system that optimizes for something that's different from what humans want. Like AI based-HFT? Which humans want wha

The worrying scenario is just any powerful system that optimizes for something that's different from what humans want. Like AI based-HFT? Which humans want what? I'm pretty sure, that if there's one constant, that there always will be some humans on either side of the argument.