4 ms·

I'm a manager at my company. We have a rating system, and there is no need for someone else to be ranked lowly for you to be ranked highly. What that DOES mea

I'm a manager at my company. We have a rating system, and there is no need for someone else to be ranked lowly for you to be ranked highly. What that DOES mean, though, is that someone has to keep pressure on people not to rank everyone as awesome. Because, look: a. Managers are, contrary to rumor, people too. We don't enjoy the awkward, painful meetings in which we deliver negative feedback. We don't enjoy telling people that they aren't getting a raise, or that they need to improve or be terminated. That sucks. Personally, it keeps me up nights, and I've heard the same from many other managers. b. Managers are also graded themselves on how effective their employees are. Like, if your employees are persistently rated below average, well, what's the common factor across your employees? Vice versa if they're all high performers. So if you just tell managers "rate your employees, follow your conscience," then there is a grade inflation incentive. Even if you're super-disciplined and make painfully realistic assessments of your employees' abilities, Bob one team over may give everyone fives, and then you end up looking like a bad manager and also maybe you get to go and put someone on a PIP. The company needs some way to stop this. Stack ranking is one such way. I don't like it, I think it's overall a bad way, but it's a mechanistic way, which is more valuable the larger your company. Small companies can have a powerful manager or small group of managers who "keep people honest." That doesn't scale. You can use another mechanic to keep people honest, but all mechanics are exploitable.

dv_dt 10y ago
So one manager is below average at hiring and randomly hires people, ending up with a nicely fit bell curve range of high and low performers. Another manager is great at hiring highly performing people, but now if they rank correctly, you look at their assessments with your cheat metric, the good manager is 'inflating' their ratings. In other words, looking at the rankings alone doesn't do anything for actually enforcing quality.
- aetherson 10y ago
  Sure. There is no single silver bullet that actually enforces quality everywhere. But look, inflation is bad. If grades aren't inflated, and I rate someone a 3, then we say, "This person is a solid employee. We're happy they're here, we're giving them an average raise, everything's cool. Do we have a little constructive feedback that says, 'Hey, here's how you could take it to the next level?' Sure. But we're happy with them." If grades are inflated and I give someone a 3, then oh shit, maybe they're in trouble! Because some asshole had to give everyone in his team a 4 or 5, and now my solid employee looks like he's underperforming the company average. The fact is, a solid 4+ rating means you're a really great employee. Teams that are all high performing are super rare, and tend to be really, really, really obvious within a company. Now, is it a valid criticism of stack ranking that it makes it not just difficult but impossible to describe a team that's solidly above average? Yes, absolutely. Stack ranking is kind of dumb. But understand that despite interesting and compelling stories about people being unfairly rated lower than they deserve, that's not the big problem that companies face. In general, the incentives for everyone is to rate people higher than they deserve, not lower (with toxic exceptions like what Fowler faced). And if you set up your whole system to make really sure that it embraces the idea of everyone being super awesome, you probably don't get a highly functional rating system out the other side. It's probably better to set one up that embraces the idea of average teams (average for your company) and let people override the system when you say, "Oh my god, this entire team is just killing it this review cycle."
  - sheepmullet 10y ago
    > Teams that are all high performing are super rare, and tend to be really, really, really obvious within a company. Teams that don't fit a normal distribution are incredibly common. And I think the bigger problem is money allocated for pay increases. If the average 3 rating employee in your team can get a 10% annualised return each time they change jobs then the company should be budgeting 10% for a grade of 3. In practice this doesn't happen.
    - sheepmullet 10y ago
      Not to mention a grade of 4 or 5 easily takes 2x the effort of a grade of 3. Yet it will get you maybe 2-5% more pay. I see lots of new devs working at way under capacity because it's just not worthwhile for them to work harder for a company.
    - Broken_Hippo 10y ago
      > Teams that don't fit a normal distribution are incredibly common. This doesn't make a team of high performing folks any less rare because extremes tend to be rare. It just means that reality doesn't always look like the average. You might have no high performers but 5 average ones, or 2 high performers and 3 below average (maybe they are just learning?). One team out of many being high performers might just be luck or a good manager, for sure, but it definitely warrants checking things out because of the rarity. The same would go for an entire team that is below par without obvious reason. It might not even be an issue with the supervising manager themselves, but with the way the company scores that sort of team.
    - aetherson 10y ago
      > Teams that don't fit a normal distribution are incredibly common. > > Now, is it a valid criticism of stack ranking that it makes it not just difficult but impossible to describe a team that's solidly above average? Yes, absolutely. Stack ranking is kind of dumb.
  - oxryly1 10y ago
    Isn't the solution to some of these problems to assess the teams results overall? If they are performing as required or expected (i.e. hitting targets) whatever the team metric, then wouldn't that explain rank inflation? The ranking of people requires either one of two things: 1. fitting your group of people to a bell curve regardless of their proper distribution in reality 2. ranking inflation or deflation Seems like a broken system designed either to generate make-work for managers and HR or to give the illusion of personnel manipulations to improve performance.
- bcrescimanno 10y ago
  One answer (not necessarily the answer) is to ensure that up-leveled calibration conversations are happening and that next-level managers are informed about their people. It's a delicate balance and, as a director myself, I don't claim to always get it right. But the goal I aim for is: if we have a good manager hiring great people and a mediocre manager hiring mediocre people that we calibrate correctly across those managers rather than claim that one is inflating their rankings.
mahyarm 10y ago
So what other mechanics would you use?
- a3n 10y ago
  At review time, get all of the group in a room, including the manager, and vote on who should leave. Because in a toxic environment, people are discussing that privately anyway, so let's be open and transparent. The person voted isn't necessarily made to leave. It's an opportunity for them to focus on improving themselves personally, possibly in a program.
  - padraic7a 10y ago
    Why would anyone have to leave? This isn't the worst idea ever, for that you'd have to include knives or acid.
  - mratzloff 10y ago
    That would encourages a lot of unworkable incentives and behaviors. It doesn't forward the goal of promoting team cohesion at all.
    - a3n 10y ago
      No, I suppose it doesn't. But neither does secret stack ranking. I once asked exactly where I was on the stack, and was told that's proprietary information.
    - prodigal_erik 10y ago
      The only way to get team cohesion is positive-sum priorities. If I spend more time helping you improve long-term and less time delivering on my own responsibilities, that needs to improve my next evaluation, otherwise I'm pretty much compelled to stop doing it.
  - koolba 10y ago
    > At review time, get all of the group in a room, including the manager, and vote on who should leave. Because in a toxic environment, people are discussing that privately anyway, so let's be open and transparent. This a genius idea. That way you push the liability of discrimination from the company to the employees at an individual level. It's like mob rule but for companies. > The person voted isn't necessarily made to leave. It's an opportunity for them to focus on improving themselves personally, possibly in a program. I can't begin to imagine how awkward it would be to be voted off the island and stick around anyway.
sbov 10y ago
Why would a manager be graded on how they grade their own people? Also, stack ranking "solves" the problem only because it makes the idea obviously ridiculous, because all managers would be graded equally.
- Broken_Hippo 10y ago
  If I read correctly, they rank yet do not have quotas. Someone performing well doesn't mean someone got scored low. In this system, a low performing person can become average by improving skills. With this in mind, it makes sense to grade the manager. If a manager has a lot of poor performers, they might not be giving staff the training, support, and feedback that is necessary to improve. If they all get high scores, they might be grading too easily. I'm guessing they have controls to help lessen this.
sebastos 10y ago
I don't really like "systems" for ranking and scoring people in general, but just as an exercise in systems design here, what about this for a strategy? 1. Each manager gives their people an absolute score 2. A meta-manager checks in with all managers over the course of the year and keeps tabs on performance of a randomly selected group of people (some scalable fraction of the total workforce. As this number goes up you could hire a few more people to do this job). 3. Each meta-manager looks for obvious goofy scoring discrepancies (how can person A have a lower rating than person B?). 4. Each manager is on the hook to defend those ratings in the face of scrutiny to justify why they rated Tony Terrible better than Andy Awesome. This expectation of accountability prevents them from inflating their scores too much. Sort of like QA for management assessments.

dv_dt 10y ago

So one manager is below average at hiring and randomly hires people, ending up with a nicely fit bell curve range of high and low performers. Another manager is great at hiring highly performing people, but now if they rank correctly, you look at their assessments with your cheat metric, the good manager is 'inflating' their ratings. In other words, looking at the rankings alone doesn't do anything for actually enforcing quality.

aetherson 10y ago
Sure. There is no single silver bullet that actually enforces quality everywhere. But look, inflation is bad. If grades aren't inflated, and I rate someone a 3, then we say, "This person is a solid employee. We're happy they're here, we're giving them an average raise, everything's cool. Do we have a little constructive feedback that says, 'Hey, here's how you could take it to the next level?' Sure. But we're happy with them." If grades are inflated and I give someone a 3, then oh shit, maybe they're in trouble! Because some asshole had to give everyone in his team a 4 or 5, and now my solid employee looks like he's underperforming the company average. The fact is, a solid 4+ rating means you're a really great employee. Teams that are all high performing are super rare, and tend to be really, really, really obvious within a company. Now, is it a valid criticism of stack ranking that it makes it not just difficult but impossible to describe a team that's solidly above average? Yes, absolutely. Stack ranking is kind of dumb. But understand that despite interesting and compelling stories about people being unfairly rated lower than they deserve, that's not the big problem that companies face. In general, the incentives for everyone is to rate people higher than they deserve, not lower (with toxic exceptions like what Fowler faced). And if you set up your whole system to make really sure that it embraces the idea of everyone being super awesome, you probably don't get a highly functional rating system out the other side. It's probably better to set one up that embraces the idea of average teams (average for your company) and let people override the system when you say, "Oh my god, this entire team is just killing it this review cycle."
- sheepmullet 10y ago
  > Teams that are all high performing are super rare, and tend to be really, really, really obvious within a company. Teams that don't fit a normal distribution are incredibly common. And I think the bigger problem is money allocated for pay increases. If the average 3 rating employee in your team can get a 10% annualised return each time they change jobs then the company should be budgeting 10% for a grade of 3. In practice this doesn't happen.
  - sheepmullet 10y ago
    Not to mention a grade of 4 or 5 easily takes 2x the effort of a grade of 3. Yet it will get you maybe 2-5% more pay. I see lots of new devs working at way under capacity because it's just not worthwhile for them to work harder for a company.
  - Broken_Hippo 10y ago
    > Teams that don't fit a normal distribution are incredibly common. This doesn't make a team of high performing folks any less rare because extremes tend to be rare. It just means that reality doesn't always look like the average. You might have no high performers but 5 average ones, or 2 high performers and 3 below average (maybe they are just learning?). One team out of many being high performers might just be luck or a good manager, for sure, but it definitely warrants checking things out because of the rarity. The same would go for an entire team that is below par without obvious reason. It might not even be an issue with the supervising manager themselves, but with the way the company scores that sort of team.
  - aetherson 10y ago
    > Teams that don't fit a normal distribution are incredibly common. > > Now, is it a valid criticism of stack ranking that it makes it not just difficult but impossible to describe a team that's solidly above average? Yes, absolutely. Stack ranking is kind of dumb.
- oxryly1 10y ago
  Isn't the solution to some of these problems to assess the teams results overall? If they are performing as required or expected (i.e. hitting targets) whatever the team metric, then wouldn't that explain rank inflation? The ranking of people requires either one of two things: 1. fitting your group of people to a bell curve regardless of their proper distribution in reality 2. ranking inflation or deflation Seems like a broken system designed either to generate make-work for managers and HR or to give the illusion of personnel manipulations to improve performance.
bcrescimanno 10y ago
One answer (not necessarily the answer) is to ensure that up-leveled calibration conversations are happening and that next-level managers are informed about their people. It's a delicate balance and, as a director myself, I don't claim to always get it right. But the goal I aim for is: if we have a good manager hiring great people and a mediocre manager hiring mediocre people that we calibrate correctly across those managers rather than claim that one is inflating their rankings.

mahyarm 10y ago

So what other mechanics would you use?

a3n 10y ago
At review time, get all of the group in a room, including the manager, and vote on who should leave. Because in a toxic environment, people are discussing that privately anyway, so let's be open and transparent. The person voted isn't necessarily made to leave. It's an opportunity for them to focus on improving themselves personally, possibly in a program.
- padraic7a 10y ago
  Why would anyone have to leave? This isn't the worst idea ever, for that you'd have to include knives or acid.
- mratzloff 10y ago
  That would encourages a lot of unworkable incentives and behaviors. It doesn't forward the goal of promoting team cohesion at all.
  - a3n 10y ago
    No, I suppose it doesn't. But neither does secret stack ranking. I once asked exactly where I was on the stack, and was told that's proprietary information.
  - prodigal_erik 10y ago
    The only way to get team cohesion is positive-sum priorities. If I spend more time helping you improve long-term and less time delivering on my own responsibilities, that needs to improve my next evaluation, otherwise I'm pretty much compelled to stop doing it.
- koolba 10y ago
  > At review time, get all of the group in a room, including the manager, and vote on who should leave. Because in a toxic environment, people are discussing that privately anyway, so let's be open and transparent. This a genius idea. That way you push the liability of discrimination from the company to the employees at an individual level. It's like mob rule but for companies. > The person voted isn't necessarily made to leave. It's an opportunity for them to focus on improving themselves personally, possibly in a program. I can't begin to imagine how awkward it would be to be voted off the island and stick around anyway.

sbov 10y ago

Why would a manager be graded on how they grade their own people? Also, stack ranking "solves" the problem only because it makes the idea obviously ridiculous, because all managers would be graded equally.

Broken_Hippo 10y ago
If I read correctly, they rank yet do not have quotas. Someone performing well doesn't mean someone got scored low. In this system, a low performing person can become average by improving skills. With this in mind, it makes sense to grade the manager. If a manager has a lot of poor performers, they might not be giving staff the training, support, and feedback that is necessary to improve. If they all get high scores, they might be grading too easily. I'm guessing they have controls to help lessen this.

sebastos 10y ago

I don't really like "systems" for ranking and scoring people in general, but just as an exercise in systems design here, what about this for a strategy? 1. Each manager gives their people an absolute score 2. A meta-manager checks in with all managers over the course of the year and keeps tabs on performance of a randomly selected group of people (some scalable fraction of the total workforce. As this number goes up you could hire a few more people to do this job). 3. Each meta-manager looks for obvious goofy scoring discrepancies (how can person A have a lower rating than person B?). 4. Each manager is on the hook to defend those ratings in the face of scrutiny to justify why they rated Tony Terrible better than Andy Awesome. This expectation of accountability prevents them from inflating their scores too much. Sort of like QA for management assessments.