This is the actual problem. We're doing Goodhart's Law with self-improvement. In a lot of cases, particularly concrerning people with power/clout, those people

This is the actual problem. We're doing Goodhart's Law with self-improvement. In a lot of cases, particularly concrerning people with power/clout, those people are never going to change. But they can make a vague simulation of it that hits the metrics. And now we've reached the stage where they can simply farm that process out to paid help, so they don't even have to do the simulation themselves.