You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible.

You need to measure the KPIs. The reason we use metrics is because things have scaled out of control, and using a "real" judgment system is no longer possible. Not for everyone, anyway. Using hiring as an example, but this should work for nearly all metrics driven workloads. Hire a small subset of your staff using a real judgment system, hire most of your staff using metrics, then take a sample of those hired using metrics and take a real look at them compared against those hired using real judgments. If they are reasonably close, the KPIs are working. If they are alien to each-other, you need to either stop using KPIs or alter them significantly to fit with what the more effective people are doing.