For one person, at one point in time, sure, even though it'd be a quite complex one. But with an AI used by a thousand companies to replace one team member each

For one person, at one point in time, sure, even though it'd be a quite complex one. But with an AI used by a thousand companies to replace one team member each, they're replacing a thousand of different "reward functions" with just one across all of them. A huge advantage of humanity is that we have different people, from different cultural and social backgrounds, all with social and personal norms and values that vary over time for each and every person, and telling an AI "imitate a pirate" or "answer without any biases" is not going to replace that.