That airline policy is the kind of micro-optimization that no individual human would do, but distributed decision-making of large organizations effect all the t

That airline policy is the kind of micro-optimization that no individual human would do, but distributed decision-making of large organizations effect all the time. There's profit to be found by targeting people in distress—like someone who just had a person close to them die—and abusing their distraction to confuse them into paying more money then they need to. Allowing a grace period to claim this "bereavement discount" is a human(e) policy. Making the client claim the discount immediately, or else lose it, is monstrous. Today, the robot showed more humanity than Air Canada's human leadership. That was an accident; the future could be the opposite of that. You could program machines to be "better" employees than humans, more aligned with organizational goals like "maximize profit at absolutely any cost", or "win wars at absolutely any cost", or "win elections at absolutely any cost". We humans aren't completely aligned with our teams; we have moral scruples that limit us—we can't achieve the 100% "absolutely any cost" part. I think that might suddenly change, and we might find ourselves drowning in an unexpectedly in-human world. (This was inspired by, I forgot who wrote it (?), a writer's observation about the evils of war being easier when morality is distributed. The commander who orders the atrocity, doesn't do it; the solider who commits the atrocity, has no agency in his actions. Both feel reduced culpability, and can go farther in effecting their goals than an individual acting alone).