I think what you are suggesting is a classic recipe for office politics. It frequently can be more difficult for a supervisor to get an accurate assessment of

I think what you are suggesting is a classic recipe for office politics. It frequently can be more difficult for a supervisor to get an accurate assessment of a direct report's skill level than a peer, both because they have many other demands on their time and because employees know it is important to put best foot forward in interaction with their boss (or, more cynically, make themselves look better than their peers/competitors). If you leave your metric fuzzy, as you suggest, managers/whoever is doing the assessment will tend towards 'gut feel' and you will get people optimizing to be their boss's best friend or to be the one who makes life easiest for them, instead of optimizing for those who are doing the best job. It isn't always this bad: individuals in manager roles can mitigate this by doing their job well, but the system overall can tend that way.