5 ms·

Noooooooo. All of these metrics are a mistake, don't measure any of them. Measure the things that actually matter about your product and whether or not it's g

Noooooooo. All of these metrics are a mistake, don't measure any of them. Measure the things that actually matter about your product and whether or not it's getting better or worse. Measure outcomes. Only outcomes. Try and make sure your metrics capture the real outcomes and not just what you see in a lab. This usually means you need to use field data to get that information.

zapita 8y ago
In larger organizations it can be very difficult for team managers to measure business outcomes of their work directly. Talking to customers is a difficult process, if at all allowed. Information from customers percolates through many layers of support and sales staff, product managers, business stakeholders, project managers etc. By the time the information gets to you it is tainted by bias and politics. Not that it would have helped you much since your objectives are already set by your hierarchy, and again are focused on your ability to execute, not business outcomes. That may not sound like a very productive environment, but that is the environmental in which most of the world’s software is written. For those engineering managers that have to operate in it, measuring your ability to execute is the best you can do. You might as well do it correctly (which is hard). And if you’re one of the lucky few working in a less dystopian environment, don’t be too quick to gloat. Enjoy it while it lasts, you’re just one acquisition or reorganization away from joining the rest of us in corporate mediocrity.
- djcapelis 8y ago
  I work for one of the world’s largest companies. If your organizational structure is a problem for connecting outcomes with engineering then it sounds like you’ve already found a great problem to solve and so go ahead and tackle that instead of spending time on bad metrics! (Full disclaimer: I am also currently working as one of those project managers you mentioned. If your project managers aren’t communicating clearly about how engineering impacts outcomes, find new project managers.)
  - zapita 8y ago
    So, based on your public profile, it appears that you work at Apple, which is not at all representative of the experience of the vast majority of software engineers. You also seem pretty young, so you’re unlikely to have a lot of experience prior to your current job. It kind of shows in your reply, too. For example you suggest that a manager should focus their efforts on changing the organization around and above them. That is very bad advice in almost every large company. First, you should focus your energy on helping your team succeed within the organization you have - not the one you wish you had. Then, if you have any bandwidth and willpower left, you can attempt modest amounts of positive change in the organization, starting with the absolute worst hair-on-fire problem you see. But keep your expectations low, because most attempts to change large organizations fail. If you are too unhappy with an organization to tolerate it, and you don’t have the power to fire the people in it, the solution is generally to quit. You appear to be one of the lucky few not in this situation, but you don’t have enough experience to realize how lucky you are - or how quickly your situation could change. You think everyone is happy with their organization at Apple? Think again. Like I said, you’re just one reorg away. Please refer to the last paragraph of my previous comment, it is extremely relevant to you.
    - djcapelis 8y ago
      It isn’t by accident that I am in a functional org. I select where I work carefully, and I walk when they aren’t functional. You should too. Managing your team is table stakes. Managing 360 isn’t a sign of lack of experience.
      - zapita 8y ago
        First you said that if I don’t like my org, I should change it. Now you’re saying that if I don’t like my org, I should quit. Which is it? Perhaps you should just stop trying to give me unsolicited career advice, and actually address the comment you’re replying to? Just a thought.
vlovich123 8y ago
The post looks reasonable to me. The point he's making is that no metric is useful for measuring developer efficacy/productivity & the underlying unstated reason is that dev works tends to be random; problem A is nothing like problem B & there's no way to objectively measure the quality of solution for problem A against another developer's solution. He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? You can't mean the final shipping SW because that's usually way too late. You want to figure out if your velocity is going to let you hit your ship date well in advance of the actual ship date so that you have time to deploy process fixes/hire more people so that you do hit the ship date.
- jeanlaf 8y ago
  Indeed. Plus, I would add that measuring the outcome (definitely a must-do, not arguing on this) is more about the product. The article was more focusing on metrics for the engineering organization, how it performs as a team. Could you say that your engineering organization is collaborating effectively, only based on the outcomes?
  - djcapelis 8y ago
    Yes. Divorcing teams from outcomes and their products is an anti-pattern. Focusing on metrics for an engineering organization not linked to the things that organization builds is harmful and will produce the wrong results. Evaluating teams on outcomes will force them to collaborate to make something great. Evaluating teams on collaboration will just make teams do whatever it is you’re measuring.
- djcapelis 8y ago
  > He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? None of the proposed metrics capture project health. Test coverage is the closest and that still isn’t great. (Unit tests are important but you can have 100% test coverage and a 0% working product. In fact, this is how every project begins before the first line of code is written.) No one gives a damn how many pull requests happen or how many days apart they are. 1) Does the software work yet? 1a) How much of it works? 2) Does it do what we want? 2a) .... everytime? Less? How many 9s? Report on the things that matter. Track the things that matter. Everything else is actively harmful and distracts your teams from doing what matters. Also, when you ship the software you haven’t finished, you’ve begun. It is absolutely still useful to keep metrics on outcomes and track improving them over time. It is absolutely not “too late” and any planning that assumes software is done after you ship it is flawed. Features can become done, software never is.

zapita 8y ago

In larger organizations it can be very difficult for team managers to measure business outcomes of their work directly. Talking to customers is a difficult process, if at all allowed. Information from customers percolates through many layers of support and sales staff, product managers, business stakeholders, project managers etc. By the time the information gets to you it is tainted by bias and politics. Not that it would have helped you much since your objectives are already set by your hierarchy, and again are focused on your ability to execute, not business outcomes. That may not sound like a very productive environment, but that is the environmental in which most of the world’s software is written. For those engineering managers that have to operate in it, measuring your ability to execute is the best you can do. You might as well do it correctly (which is hard). And if you’re one of the lucky few working in a less dystopian environment, don’t be too quick to gloat. Enjoy it while it lasts, you’re just one acquisition or reorganization away from joining the rest of us in corporate mediocrity.

djcapelis 8y ago
I work for one of the world’s largest companies. If your organizational structure is a problem for connecting outcomes with engineering then it sounds like you’ve already found a great problem to solve and so go ahead and tackle that instead of spending time on bad metrics! (Full disclaimer: I am also currently working as one of those project managers you mentioned. If your project managers aren’t communicating clearly about how engineering impacts outcomes, find new project managers.)
- zapita 8y ago
  So, based on your public profile, it appears that you work at Apple, which is not at all representative of the experience of the vast majority of software engineers. You also seem pretty young, so you’re unlikely to have a lot of experience prior to your current job. It kind of shows in your reply, too. For example you suggest that a manager should focus their efforts on changing the organization around and above them. That is very bad advice in almost every large company. First, you should focus your energy on helping your team succeed within the organization you have - not the one you wish you had. Then, if you have any bandwidth and willpower left, you can attempt modest amounts of positive change in the organization, starting with the absolute worst hair-on-fire problem you see. But keep your expectations low, because most attempts to change large organizations fail. If you are too unhappy with an organization to tolerate it, and you don’t have the power to fire the people in it, the solution is generally to quit. You appear to be one of the lucky few not in this situation, but you don’t have enough experience to realize how lucky you are - or how quickly your situation could change. You think everyone is happy with their organization at Apple? Think again. Like I said, you’re just one reorg away. Please refer to the last paragraph of my previous comment, it is extremely relevant to you.
  - djcapelis 8y ago
    It isn’t by accident that I am in a functional org. I select where I work carefully, and I walk when they aren’t functional. You should too. Managing your team is table stakes. Managing 360 isn’t a sign of lack of experience.
    - zapita 8y ago
      First you said that if I don’t like my org, I should change it. Now you’re saying that if I don’t like my org, I should quit. Which is it? Perhaps you should just stop trying to give me unsolicited career advice, and actually address the comment you’re replying to? Just a thought.

vlovich123 8y ago

The post looks reasonable to me. The point he's making is that no metric is useful for measuring developer efficacy/productivity & the underlying unstated reason is that dev works tends to be random; problem A is nothing like problem B & there's no way to objectively measure the quality of solution for problem A against another developer's solution. He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? You can't mean the final shipping SW because that's usually way too late. You want to figure out if your velocity is going to let you hit your ship date well in advance of the actual ship date so that you have time to deploy process fixes/hire more people so that you do hit the ship date.

jeanlaf 8y ago
Indeed. Plus, I would add that measuring the outcome (definitely a must-do, not arguing on this) is more about the product. The article was more focusing on metrics for the engineering organization, how it performs as a team. Could you say that your engineering organization is collaborating effectively, only based on the outcomes?
- djcapelis 8y ago
  Yes. Divorcing teams from outcomes and their products is an anti-pattern. Focusing on metrics for an engineering organization not linked to the things that organization builds is harmful and will produce the wrong results. Evaluating teams on outcomes will force them to collaborate to make something great. Evaluating teams on collaboration will just make teams do whatever it is you’re measuring.
djcapelis 8y ago
> He's instead proposing metrics that you can use as a high level barometer of project health. What outcome are you looking for? None of the proposed metrics capture project health. Test coverage is the closest and that still isn’t great. (Unit tests are important but you can have 100% test coverage and a 0% working product. In fact, this is how every project begins before the first line of code is written.) No one gives a damn how many pull requests happen or how many days apart they are. 1) Does the software work yet? 1a) How much of it works? 2) Does it do what we want? 2a) .... everytime? Less? How many 9s? Report on the things that matter. Track the things that matter. Everything else is actively harmful and distracts your teams from doing what matters. Also, when you ship the software you haven’t finished, you’ve begun. It is absolutely still useful to keep metrics on outcomes and track improving them over time. It is absolutely not “too late” and any planning that assumes software is done after you ship it is flawed. Features can become done, software never is.