6 ms·

In my experience, metrics are a giant waste of time for engineers and engineering managers. Project managers and folks who depend on the fruits of engineering l

In my experience, metrics are a giant waste of time for engineers and engineering managers. Project managers and folks who depend on the fruits of engineering labor on the other hand love these metrics. It is simpler (and wrongheaded) because who wants complications after all? Look at the trivialization in this article for instance - "tasks come in, and software comes out". Oh really? I had no idea! Tasks vary drastically from project to project,including due to varying requirements, ambient software environment, technological changes and a host of other factors. Metrics are a way for project managers to pass the buck on to the engineering teams and nothing more.

tra3 4y ago
You gotta have some way of answering the "how long is it going to take" question. There's no better way of predicting things than looking at how long things have taken in the past. Unfortunately what everyone forgets is that historical information only applies if you are doing "a thing" that is similar to what you've done before.
- djbusby 4y ago
  Q: how long will this take? A: what is your budget? What other work will you sacrifice?
- tadfisher 4y ago
  One way to address this problem is to not ask the question, but to work with your teams to set goals for a longer period of time such as a quarter. Then structure your organization such that it doesn't depend on the exact timing of work completion.
  - cortesoft 4y ago
    How does a longer period of time help with estimation? Trying to estimate what you will complete in a quarter is even harder than estimating what you will complete in a week… errors compound.
    - AdieuToLogic 4y ago
      How does a longer period of time help with estimation? I interpreted the GP's statement of: One way to address this problem is to not ask the question, but to work with your teams to set goals for a longer period of time ... To mean, "don't focus on estimating each individual task, but instead set midrange objectives." Which contextualizes the GP's suggestion of: Then structure your organization such that it doesn't depend on the exact timing of work completion. To imply avoid the desire to "estimate what you will complete in a quarter" and instead focus on delivering prioritized stakeholder value.
      - cortesoft 4y ago
        If you are setting “midrange objectives” for a quarter, you are basically saying “I estimate I will complete this much work in the quarter”, because you are estimating you can complete enough work to meet the objective in a quarter. Either the objective is easy enough that the estimate doesn’t have to be that accurate to still meet it, or the objective is not specific enough to require an accurate estimate. I actually mostly agree with the comment I was replying to, I just think the “set goals for a longer period of time” was not filling in the whole picture. You can’t just set the same types of goals you had when you were making sprint estimations, stretch them out to be quarter estimations, and expect the problem to go away.
        AdieuToLogic 4y ago
        I think the difference is in plurality. Setting goals for a quarter (or month, or year) to me is more of a team thing/focus/discussion rather than an individual's estimation of what they can do. And I completely agree with you that stretching a sprint into a quarter makes no sense if it is only for estimating one person's work. Given that so much of what we do as developers typically depends on more than just ourself in an effort, I agree with what I think tadfisher is expressing; realistic delivery planning can be achieved by focusing on "teams and goals for the next few months" and cannot be by trying to answer "how long will <insert feature never done before> take?"
- switchbak 4y ago
  It really comes down to: can you provide the value we need in this area. That value might be "validate experimental research is implementable", or "perform a spike to de-risk a future effort". There's a fog of uncertainty that gets impenetrable past a certain time horizon. We need businesses that are capable of forging ahead into the unknown, being honest about the level of confidence given our knowledge at the time. I don't see that approach coming out of MBA schools, and I think Elon's approach is in marked contrast to that. In many ways old school grassroots agile has penetrated the lower levels of the org (to some degree), but the upper levels haven't changed in decades, and this impedance mismatch is most visible in the rigidity of planning (which is also forced on them by outside pressures).
- deleted 4y ago
  [deleted]
- coffeemug 4y ago
  You don't ask "how long is it going to take?" You set a deadline and let the black box do whatever internal negotiations it needs to meet it.
  - fooster 4y ago
    Imagine if flight planning worked like that. We are going to get to Paris from New York by 8. Instead the plane lands in the ocean.
  - tra3 4y ago
    That's bleak. Where does the black box start? CEO to CTO? CTO to VP of Engineering? VP to Director? Director to Team Lead?
- ip26 4y ago
  Or, even more difficult, are you on track to deliver your twelve month goals? Hard to know, and easy to lie to yourself.
- doctor_eval 4y ago
  > You gotta have some way of answering the "how long is it going to take" question. Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete. While estimation seems to be a shibboleth of certain types of project management, the fact that implementation time estimates are rarely accurate - and often significantly wrong - suggests that they are not as vital as we’re lead to believe.
  - tra3 4y ago
    > While estimation seems to be a shibboleth of certain types of project management, the fact that implementation time estimates are rarely accurate - and often significantly wrong - suggests that they are not as vital as we’re lead to believe. I disagree with your conclusion. Just because we're terrible at estimating doesn't mean it's not important. "100% uptime is rarely achieved, which means it's unnecessary and unimportant". > Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete. McConnel talks about it in "Software Estimation". The only way to accurately predict anything is to base it on your previous experience, which I allude to. Unfortunately (or fortunately, depending on how you look at it), even projects that looks similar on the surface are more frequently dissimilar than similar. Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something.
    - doctor_eval 4y ago
      I understand that we disagree, but Eric Evans also talks about how implementation is an inescapable part of design, as does Jack Reeves. So it follows, for me at least, that estimates can never be accurate. Recently posted on HN was a link to a post at Lunar Logic where estimation is basically broken into 3 categories: “NFC” (no clue), “TFB” (too big) or “1”. They explain why here [1]. This has been my experience over more than 30 years of building products. The duration of any given feature/story is effectively unknowable, but we can make a decent guess as to it’s achievability, we can get a decent idea of the rate at which stories are completed. The main problem I have with more detailed estimation is that the estimation process itself consumes significant engineering resources which IMO should be assigned to actual engineering. To your specific points: > "100% uptime is rarely achieved, which means it's unnecessary and unimportant". This is specious. As an industry our estimates tend to be wildly wrong, and despite decades of dealing with “the software crisis” they are not improving. My position is that this means that we’re spending a lot of time estimating and then more time dealing with them being wrong. It’s just a waste. > Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something. This seems to be the wrong way to do it. Surely the approach should be to ask what value this “something” has to the business, and then determine if it’s technically feasible within that value. But that would mean more work for the executives, and quite frankly this, in my direct experience, is not the kind of responsibility they like to take on. [1] https://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-logic/ https://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-log...
bergenty 4y ago
The hard truth is metrics work. I can quantify someone’s performance and if they’re not doing well put pressure on them to do better or hire someone else. Velocity is a pretty good measure of someone’s performance especially if the estimating is done in a sprint planning session with the entire team.
- jdlshore 4y ago
  The harder truth is that metrics only appear to work. As soon as you use metrics to judge performance, people will start gaming them. They’ll do whatever it takes to get a good score, regardless of whether it’s good for the company or its customers. The metrics “work,” in that they’ll go up, but the things you don’t measure will get worse, often (eventually) catastrophically. In the case of velocity, you’ll get people taking shortcuts and sacrificing quality, both internal and external, so they can juice their numbers. The outcome is technical debt and arguments about what it means for something to be done, resulting in slower progress overall. Source: I’ve been consulting in this space for a few decades now, and have seen the consequences of using velocity as a performance measure time and time again. (Other metrics are just as bad. See Robert Austin, Measuring and Managing Performance in Organizations, for an explanation of why knowledge work like software development is incompletely measurable and thus subject to measurement dysfunction.)
  - mlyle 4y ago
    The truth is somewhere in between. Metrics are great to diagnose the overall process and get a sense of what's going on that can be superior to our qualitative feels about it. And metrics can also spot an occasional outlier or performance problem. Used sparingly, this does not encourage juicing the numbers.
    - jdlshore 4y ago
      Using data and metrics for analysis and self-reflection is great, when used thoughtfully. The problems arise when they’re used to judge performance—or even perceived to be used to judge performance. That’s why they’re so tricky to use well. You have to set up a situation where it’s systematically impossible to abuse metrics, typically by putting the data and analysis/judgement at the same level, and only reporting aggregated/anonymized results and qualitative conclusions rather than the raw data. Some people don’t know how to manage without measurements, punishments, and rewards. It’s a correctable flaw. Measurement-based management is called “theory X” management, but knowledge work needs “theory Y” management. There’s a lot of material out there on how to do it, including a section on it in my book.
throwaway787544 4y ago
I am an engineer, and I rely heavily on metrics, because my job requires quantifying that a system is working correctly. It is impossible to do that correctly without metrics. Similarly, managing a product team is also managing a running system; that system is just made up of meat sacks whacking plastic buttons with their bony protrusions, rather than computers humming away in data centers. A product manager still needs to quantify that the system making that product is working correctly, and metrics are essential. Otherwise you will only be guessing as to how the system is working, and those guesses will be much more haphazard as the system grows.

tra3 4y ago

You gotta have some way of answering the "how long is it going to take" question. There's no better way of predicting things than looking at how long things have taken in the past. Unfortunately what everyone forgets is that historical information only applies if you are doing "a thing" that is similar to what you've done before.

djbusby 4y ago
Q: how long will this take? A: what is your budget? What other work will you sacrifice?
tadfisher 4y ago
One way to address this problem is to not ask the question, but to work with your teams to set goals for a longer period of time such as a quarter. Then structure your organization such that it doesn't depend on the exact timing of work completion.
- cortesoft 4y ago
  How does a longer period of time help with estimation? Trying to estimate what you will complete in a quarter is even harder than estimating what you will complete in a week… errors compound.
  - AdieuToLogic 4y ago
    How does a longer period of time help with estimation? I interpreted the GP's statement of: One way to address this problem is to not ask the question, but to work with your teams to set goals for a longer period of time ... To mean, "don't focus on estimating each individual task, but instead set midrange objectives." Which contextualizes the GP's suggestion of: Then structure your organization such that it doesn't depend on the exact timing of work completion. To imply avoid the desire to "estimate what you will complete in a quarter" and instead focus on delivering prioritized stakeholder value.
    - cortesoft 4y ago
      If you are setting “midrange objectives” for a quarter, you are basically saying “I estimate I will complete this much work in the quarter”, because you are estimating you can complete enough work to meet the objective in a quarter. Either the objective is easy enough that the estimate doesn’t have to be that accurate to still meet it, or the objective is not specific enough to require an accurate estimate. I actually mostly agree with the comment I was replying to, I just think the “set goals for a longer period of time” was not filling in the whole picture. You can’t just set the same types of goals you had when you were making sprint estimations, stretch them out to be quarter estimations, and expect the problem to go away.
      - AdieuToLogic 4y ago
        I think the difference is in plurality. Setting goals for a quarter (or month, or year) to me is more of a team thing/focus/discussion rather than an individual's estimation of what they can do. And I completely agree with you that stretching a sprint into a quarter makes no sense if it is only for estimating one person's work. Given that so much of what we do as developers typically depends on more than just ourself in an effort, I agree with what I think tadfisher is expressing; realistic delivery planning can be achieved by focusing on "teams and goals for the next few months" and cannot be by trying to answer "how long will <insert feature never done before> take?"
switchbak 4y ago
It really comes down to: can you provide the value we need in this area. That value might be "validate experimental research is implementable", or "perform a spike to de-risk a future effort". There's a fog of uncertainty that gets impenetrable past a certain time horizon. We need businesses that are capable of forging ahead into the unknown, being honest about the level of confidence given our knowledge at the time. I don't see that approach coming out of MBA schools, and I think Elon's approach is in marked contrast to that. In many ways old school grassroots agile has penetrated the lower levels of the org (to some degree), but the upper levels haven't changed in decades, and this impedance mismatch is most visible in the rigidity of planning (which is also forced on them by outside pressures).
deleted 4y ago
[deleted]
coffeemug 4y ago
You don't ask "how long is it going to take?" You set a deadline and let the black box do whatever internal negotiations it needs to meet it.
- fooster 4y ago
  Imagine if flight planning worked like that. We are going to get to Paris from New York by 8. Instead the plane lands in the ocean.
- tra3 4y ago
  That's bleak. Where does the black box start? CEO to CTO? CTO to VP of Engineering? VP to Director? Director to Team Lead?
ip26 4y ago
Or, even more difficult, are you on track to deliver your twelve month goals? Hard to know, and easy to lie to yourself.
doctor_eval 4y ago
> You gotta have some way of answering the "how long is it going to take" question. Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete. While estimation seems to be a shibboleth of certain types of project management, the fact that implementation time estimates are rarely accurate - and often significantly wrong - suggests that they are not as vital as we’re lead to believe.
- tra3 4y ago
  > While estimation seems to be a shibboleth of certain types of project management, the fact that implementation time estimates are rarely accurate - and often significantly wrong - suggests that they are not as vital as we’re lead to believe. I disagree with your conclusion. Just because we're terrible at estimating doesn't mean it's not important. "100% uptime is rarely achieved, which means it's unnecessary and unimportant". > Code (implementation) is an integral alert part of the software design and planning process, and you can’t know how long something will take until that’s complete. McConnel talks about it in "Software Estimation". The only way to accurately predict anything is to base it on your previous experience, which I allude to. Unfortunately (or fortunately, depending on how you look at it), even projects that looks similar on the surface are more frequently dissimilar than similar. Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something.
  - doctor_eval 4y ago
    I understand that we disagree, but Eric Evans also talks about how implementation is an inescapable part of design, as does Jack Reeves. So it follows, for me at least, that estimates can never be accurate. Recently posted on HN was a link to a post at Lunar Logic where estimation is basically broken into 3 categories: “NFC” (no clue), “TFB” (too big) or “1”. They explain why here [1]. This has been my experience over more than 30 years of building products. The duration of any given feature/story is effectively unknowable, but we can make a decent guess as to it’s achievability, we can get a decent idea of the rate at which stories are completed. The main problem I have with more detailed estimation is that the estimation process itself consumes significant engineering resources which IMO should be assigned to actual engineering. To your specific points: > "100% uptime is rarely achieved, which means it's unnecessary and unimportant". This is specious. As an industry our estimates tend to be wildly wrong, and despite decades of dealing with “the software crisis” they are not improving. My position is that this means that we’re spending a lot of time estimating and then more time dealing with them being wrong. It’s just a waste. > Regardless it's useful to be able to answer the "how long" question at least a coarse level to see if it's worth doing something. This seems to be the wrong way to do it. Surely the approach should be to ask what value this “something” has to the business, and then determine if it’s technically feasible within that value. But that would mean more work for the executives, and quite frankly this, in my direct experience, is not the kind of responsibility they like to take on. [1] https://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-logic/ https://blog.lunarlogic.io/2016/how-we-estimate-at-lunar-log...

bergenty 4y ago

The hard truth is metrics work. I can quantify someone’s performance and if they’re not doing well put pressure on them to do better or hire someone else. Velocity is a pretty good measure of someone’s performance especially if the estimating is done in a sprint planning session with the entire team.

jdlshore 4y ago
The harder truth is that metrics only appear to work. As soon as you use metrics to judge performance, people will start gaming them. They’ll do whatever it takes to get a good score, regardless of whether it’s good for the company or its customers. The metrics “work,” in that they’ll go up, but the things you don’t measure will get worse, often (eventually) catastrophically. In the case of velocity, you’ll get people taking shortcuts and sacrificing quality, both internal and external, so they can juice their numbers. The outcome is technical debt and arguments about what it means for something to be done, resulting in slower progress overall. Source: I’ve been consulting in this space for a few decades now, and have seen the consequences of using velocity as a performance measure time and time again. (Other metrics are just as bad. See Robert Austin, Measuring and Managing Performance in Organizations, for an explanation of why knowledge work like software development is incompletely measurable and thus subject to measurement dysfunction.)
- mlyle 4y ago
  The truth is somewhere in between. Metrics are great to diagnose the overall process and get a sense of what's going on that can be superior to our qualitative feels about it. And metrics can also spot an occasional outlier or performance problem. Used sparingly, this does not encourage juicing the numbers.
  - jdlshore 4y ago
    Using data and metrics for analysis and self-reflection is great, when used thoughtfully. The problems arise when they’re used to judge performance—or even perceived to be used to judge performance. That’s why they’re so tricky to use well. You have to set up a situation where it’s systematically impossible to abuse metrics, typically by putting the data and analysis/judgement at the same level, and only reporting aggregated/anonymized results and qualitative conclusions rather than the raw data. Some people don’t know how to manage without measurements, punishments, and rewards. It’s a correctable flaw. Measurement-based management is called “theory X” management, but knowledge work needs “theory Y” management. There’s a lot of material out there on how to do it, including a section on it in my book.

throwaway787544 4y ago

I am an engineer, and I rely heavily on metrics, because my job requires quantifying that a system is working correctly. It is impossible to do that correctly without metrics. Similarly, managing a product team is also managing a running system; that system is just made up of meat sacks whacking plastic buttons with their bony protrusions, rather than computers humming away in data centers. A product manager still needs to quantify that the system making that product is working correctly, and metrics are essential. Otherwise you will only be guessing as to how the system is working, and those guesses will be much more haphazard as the system grows.