Some managers might grade their employees more harshly, and some might be soft touches. Within a departmental level, it may be easier for a manager to argue tha

Some managers might grade their employees more harshly, and some might be soft touches. Within a departmental level, it may be easier for a manager to argue that his team were all stars, but how do you deal with the question of an entire department using a harsher or more lenient grading system? I think there needs to be a small group of senior engineers -- not managers, or at least, not just managers -- who understand what each team is doing well enough to have a feel for the team's overall competence. Their ratings of the teams can then contextualize the teams' ratings of their members. Over the long term, these team ratings can be checked by occasionally trading people between teams -- if a middle member of one team becomes a star in another, that suggests a difference in team levels. A single such observation is not a perfect indicator because individual skills and interests also play a role, but with a number of such experiments over time I think one can get a pretty good picture. "Make everything as simple as possible, but no simpler." The assumption that the teams all have the same average competence is a simplifying assumption, to be sure -- but it makes things too simple.