7 ms·

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review

by yodon 4y ago

My favorite quantitative metrics for engineering teams: - Avg time from code review requested to code review picked up - Avg time to complete code review - Avg time from eng done to first customer using it - Avg time from eng done to full production release - Fraction of tasks started that never reach a customer These are loosely based on the Japanese concept of Muda (waste), as personified in the physical logistics world via the acronym TIM WOOD (or TIM WOODS)[0] and are similar but not identical to the DORA metrics. The time to complete a code review is there (for example) not to focus on the amount of time actually spent performing the code review but to focus on all the waiting around the actual code review, which is typically much much longer than the time spent doing the review itself. It's not uncommon to see organizations where an engineer will submit code for review and then have to wait a day or more for someone to pick up their request and then another day or more for that other person to get around to reviewing it. If there are comments on the commit that need to be responded to, you can see additional delays. These "minor inefficiencies" can have huge impacts on the poor dev who is trying to get their code merged, and cumulatively they result in significant increases in feature latency, the total calendar time required to ship a feature. [0] https://www.shmula.com/28695-2/28695/ https://www.shmula.com/28695-2/28695/

kqr 4y ago
These are great! How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable. In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to use it. I would even say that canceling many started tasks is directly correlated with a quick cycle time, by Little's law.
- yodon 4y ago
  > Why are you using the last one? The last one is definitely in a different bucket for me than the first four. For starters, all of these are there primarily to encourage conversation. That said, the first four can make a lot more sense to try to graph and track and optimize. The last one tends to be more purely about driving a conversation. At the top level, if you're genuinely doing a ton of learning along the way to shipping the right feature to the customer, then arguably that value is ultimately reaching the customer. If on the other hand you can't decide what the goal is and you keep changing your mind (as is often the case), then you tend to end up with a lot of dev investment made in things that simply never ship. It's also worth differentiating technical "spikes" from feature "experiments." In my vocabulary, spikes are things where you're internally assessing a question like "could we do this" and experiments are things where you're externally assessing "do customers want this/does this have the impact we want." If you have a lot of experiments that don't reach customers, you're burning a lot of dev time on things that aren't actually experiments (because by this definition experiments need to reach the customer surface to deliver data). That's a signal you should probably be looking at. Spikes generally only reach the customer indirectly (through an eventual shipping feature), but if you have a lot of spikes that don't ever reach the customer in any way that's also a signal you should probably be looking at.
- district487 4y ago
  not the OP, but I would use it as a leading indicator that there is something wrong with the way we're scoping and prioritizing tasks. If 100% of our tasks are no longer viable, it's a signal that we're doing something fundamentally wrong earlier in the chain, e.g. are we misunderstanding the user problems? the market? the wrong prioritization? not enough scoping done?
Brian_K_White 4y ago
What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous process and I say a mistake to try to make it one. Edit to address a sense I'm getting from several comments at once, that I may be a manager who doesn't understand or care about developers: I AM the developer (& sysadmin) with a bunch of fresh and stale projects and it doesn't bother me at all. I always have work to do but it's not stressfull because that is about your boss(es) not about the number of open items. If anyone tried to say all that stuff needs to be done yesterday, sure that would be a problem. But all they really are is a spectrum of priorities. Some are just ideas little more than stubs for possible future interest, some would be nice but may never be justifiable in a strict easily quanifiable sense, you can't bill any customer for the hours, and yet would still be nice, would be one of the things that sets your product apart and attracts the customer in the first place, and some have normal priority, and when something high priority comes along, it simply displaces the rest. It just requires a boss who doesn't say that every single new item is always the most important. That stack of available projects, and the age of half of them, doesn't bother me at all. Half the time, a different project is as good as a walk in the park for the reset / fresh look factor when stuck on something.
- kqr 4y ago
  This is a common conflict! One perspective says that idle engineers cost money, so you should load up your engineers to 99 % utilisation with a huge backlog of tasks. The consequence of this is that tasks take, on average, months to complete because they sit idle most of the time, not being worked on. The other perspective is that idle tasks is what's expensive. Thus you must keep your engineers lightly loaded (say 70 %) so they can, in your words, pounce on new tasks. With this setup, most tasks spend a negligible time waiting and most time actively worked on. Now, how can idle tasks be more expensive than idle engineers? There are whole books dedicated to explaining this. I'd recommend starting with Reinertsen's Principles of Product Development Flow.
kqr 4y ago
I use a similar set: - Time between idea first recorded and idea in production. - Time between idea merged to main branch and idea in production (in some places this is negligible, in others where ideas spend most of their time hanging out -- done, but collecting dust). - Time actively spent on task over the time between work started on idea and idea merged to main branch. (This is "flow efficiency" in lean terminology.) Other than that I use the DORA metrics too. I measure defect rate by looking at proportion of merge requests that fix defects, because mostly that is a stable distribution.
dilyevsky 4y ago
So just rubberstamp your pr with a nit to score awesome on the first two?
- hatware 4y ago
  You can game any system. I think the point is to give yourself guardrails that help you make good decisions for the present and future.
  - dilyevsky 4y ago
    Wouldn’t even require bad actors - people will naturally er on the side of stamp and forget if you announce this metric as goal
- yodon 4y ago
  > So just rubberstamp your pr with a nit to score awesome on the first two? If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers. If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the system in exactly the way it's designed to be "gamed." In general, if devs are talking to each other and aware of what each other is doing, efficiencies are high and latencies are short. If no one is talking to each other, efficiencies are low and latencies are high. Talking to each other and coordinating with each other is a good thing.
  - dilyevsky 4y ago
    So you’re becoming “aware of what each other is doing” by half reading the pr title and going straight to approve button? I think “legitimately “ is doing a lot of lifting here too
    - yodon 4y ago
      You seem to presuppose that the people involved in the team conversations about these metrics are stupid or clueless to a point of willfulness or intentionally bad actors. If that's the case, as with the case where you're somehow able to approve your own CR's, you face problems unrelated to the numbers. If those are the kind of problems you face, you need to fix them well before you try to bring in any processes designed to help a well functioning organization function better, including the case you called out where devs are approving code reviews without understanding them.
feoren 4y ago
> - Avg time to complete code review Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.
- yodon 4y ago
  > Doesn't [Code Review time] depend heavily on the size of each PR? The amount of active "eyeballs on code" time to complete a code review can of course vary tremendously. That said, when I see code reviews taking a week to complete, only very very very rarely is that a sign that developers are routinely spending a solid week of eyeballs-on-code time completing those code reviews. Most of the time it involves a relatively small amount of focused review time and a huge amount of not actually getting around to starting doing the code review time. If everything in your shop is so well handled that you're not wasting any time waiting for code to get reviewed, in practice your shop is probably also already making good choices about how to structure code for efficient reviews as well (but that's more an anecdotal observation about correlations, not directly an observation about causality).
- kqr 4y ago
  Have you measured this in your organisation? In my experience, every team has a fairly distinct cut-off point where if you make PRs larger than that, they start to hang around in someone's review queue forever. Every time I've made a larger PR I've regretted it and subsequently split it up into smaller ones. I get higher quality feedback and faster feedback to boot that way.
  - dkersten 4y ago
    > if you make PRs larger than that, they start to hang around in someone's review queue forever. Or you just get a "ship it" without more than a cursory glance.
- mdoms 4y ago
  Large PRs are one of the most serious anti patterns in day to day work.
puchatek 4y ago
Just the fact that these are team-oriented and not individual-oriented makes them above average valuable already
marcosdumay 4y ago
> Fraction of tasks started that never reach a customer If you optimize this down, you punish any kind of innovative or ambitious task.
- lukeramsden 4y ago
  Depends how you measure "reaching", i.e. all refactorings and optimisations all reach the customer technically "reach" the customer as soon as they're deployed (essentially, anything that meets its definition-of-done is "delivered). I think the wider idea is to measure work being interrupted and abandoned, or being abandoned due to not being necessary, both of which could be dubbed signs of inefficiency.
  - marcosdumay 4y ago
    Hum, no. I'm talking exactly about abandoned work. If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway. And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough view to know if your software is such one.
    - yodon 4y ago
      In practice I generally see the abandoned work fraction dominated by much less interesting and much less valuable poor planning decisions, false starts, and misalignment between teams. Genuine high-value risk taking is typically already so rare in most organizations that it doesn't even show up significantly in a count like this. Equally importantly, when those genuine risk taking activities are terminated they typically happen in the form of an experiment that reached a customer at some level as part of the decision to terminate the effort. If it reached a customer in some manner like that it's not in the bucket of tasks started that never reached a customer.
    - jrockway 4y ago
      This is worth a read: https://apenwarr.ca/log/20171213 https://apenwarr.ca/log/20171213 Specifically, look at the "PM can't make up their mind" graphs (https://apenwarr.ca/log/20171213#slide13 https://apenwarr.ca/log/20171213#slide13). That's the state that you're trying to avoid. I think it's fine to have tasks like "investigate the feasibility of X" and finding it infeasible. To me, that outcome is similar to delivering a feature (your customers will pay less for that than an actual feature, though, probably). What you want to avoid is "do X at any cost", and then the first steps are "integrate with third party service Y" "get a support contract for service Y" "do a giant refactoring to better use service Y", and then finding that X is actually infeasible. Now you have complexity, cost, and wasted months without having anything to give to the user. (Conflict of interest note: go ahead and do that. More work that never ships = more developers needed = higher pay for developers!)
splittingTimes 4y ago
They sure are nice, but how do you collect that data in an automated fashion?
- rsanek 4y ago
  Top two should be available through an ETL of whatever VCS is used. Next two depend on how fine-grained you want to be--either you don't automate because you stay high-level on the order of products, or you go lower-level and use feature flags or even basic logging for when a given feature is used. Not sure how you easily measure the last one though.
- kqr 4y ago
  Obvious follow-up: why do you need to automate it? You get decent precision by sampling just 10 % of what's going on. Depending on team size, that's manageable by just one person spending half an hour on it here and there. And in the process of asking questions and poking around this person learns a lot about what's happening.

by yodon 4y ago

kqr 4y ago

These are great! How are you using the last one? I feel like fraction of tasks started that reach customer could easily become misleading: you want to quickly abandon tasks once you realise they're no longer viable. In fact, the development effort should be partially about finding reasons to stop working on the thing, so you can toss it out as soon as possible, instead of waiting for the customer to turn out not to use it. I would even say that canceling many started tasks is directly correlated with a quick cycle time, by Little's law.

yodon 4y ago
> Why are you using the last one? The last one is definitely in a different bucket for me than the first four. For starters, all of these are there primarily to encourage conversation. That said, the first four can make a lot more sense to try to graph and track and optimize. The last one tends to be more purely about driving a conversation. At the top level, if you're genuinely doing a ton of learning along the way to shipping the right feature to the customer, then arguably that value is ultimately reaching the customer. If on the other hand you can't decide what the goal is and you keep changing your mind (as is often the case), then you tend to end up with a lot of dev investment made in things that simply never ship. It's also worth differentiating technical "spikes" from feature "experiments." In my vocabulary, spikes are things where you're internally assessing a question like "could we do this" and experiments are things where you're externally assessing "do customers want this/does this have the impact we want." If you have a lot of experiments that don't reach customers, you're burning a lot of dev time on things that aren't actually experiments (because by this definition experiments need to reach the customer surface to deliver data). That's a signal you should probably be looking at. Spikes generally only reach the customer indirectly (through an eventual shipping feature), but if you have a lot of spikes that don't ever reach the customer in any way that's also a signal you should probably be looking at.
district487 4y ago
not the OP, but I would use it as a leading indicator that there is something wrong with the way we're scoping and prioritizing tasks. If 100% of our tasks are no longer viable, it's a signal that we're doing something fundamentally wrong earlier in the chain, e.g. are we misunderstanding the user problems? the market? the wrong prioritization? not enough scoping done?

Brian_K_White 4y ago

What do I care about start to finish time? The only way to shorten that latency would be to have so many employees that they can all spend all day just waiting to pounce immediately on their inbox. It's just the same or worse waste distributed differently. A developer with a full pipeline seems the most efficient to me. Let there be 15 projects all in different stages of progress. It's fundamentally not a synchronous process and I say a mistake to try to make it one. Edit to address a sense I'm getting from several comments at once, that I may be a manager who doesn't understand or care about developers: I AM the developer (& sysadmin) with a bunch of fresh and stale projects and it doesn't bother me at all. I always have work to do but it's not stressfull because that is about your boss(es) not about the number of open items. If anyone tried to say all that stuff needs to be done yesterday, sure that would be a problem. But all they really are is a spectrum of priorities. Some are just ideas little more than stubs for possible future interest, some would be nice but may never be justifiable in a strict easily quanifiable sense, you can't bill any customer for the hours, and yet would still be nice, would be one of the things that sets your product apart and attracts the customer in the first place, and some have normal priority, and when something high priority comes along, it simply displaces the rest. It just requires a boss who doesn't say that every single new item is always the most important. That stack of available projects, and the age of half of them, doesn't bother me at all. Half the time, a different project is as good as a walk in the park for the reset / fresh look factor when stuck on something.

kqr 4y ago
This is a common conflict! One perspective says that idle engineers cost money, so you should load up your engineers to 99 % utilisation with a huge backlog of tasks. The consequence of this is that tasks take, on average, months to complete because they sit idle most of the time, not being worked on. The other perspective is that idle tasks is what's expensive. Thus you must keep your engineers lightly loaded (say 70 %) so they can, in your words, pounce on new tasks. With this setup, most tasks spend a negligible time waiting and most time actively worked on. Now, how can idle tasks be more expensive than idle engineers? There are whole books dedicated to explaining this. I'd recommend starting with Reinertsen's Principles of Product Development Flow.

kqr 4y ago

I use a similar set: - Time between idea first recorded and idea in production. - Time between idea merged to main branch and idea in production (in some places this is negligible, in others where ideas spend most of their time hanging out -- done, but collecting dust). - Time actively spent on task over the time between work started on idea and idea merged to main branch. (This is "flow efficiency" in lean terminology.) Other than that I use the DORA metrics too. I measure defect rate by looking at proportion of merge requests that fix defects, because mostly that is a stable distribution.

dilyevsky 4y ago

So just rubberstamp your pr with a nit to score awesome on the first two?

hatware 4y ago
You can game any system. I think the point is to give yourself guardrails that help you make good decisions for the present and future.
- dilyevsky 4y ago
  Wouldn’t even require bad actors - people will naturally er on the side of stamp and forget if you announce this metric as goal
yodon 4y ago
> So just rubberstamp your pr with a nit to score awesome on the first two? If you're saying you can CR your own commits, that's a different problem to discuss unrelated to the numbers. If you're saying you're coordinating so closely with the other devs on your team that they're picking up your CR's immediately and they're able to legitimately approve them with just a trivial nit, then congrats - you're "gaming" the system in exactly the way it's designed to be "gamed." In general, if devs are talking to each other and aware of what each other is doing, efficiencies are high and latencies are short. If no one is talking to each other, efficiencies are low and latencies are high. Talking to each other and coordinating with each other is a good thing.
- dilyevsky 4y ago
  So you’re becoming “aware of what each other is doing” by half reading the pr title and going straight to approve button? I think “legitimately “ is doing a lot of lifting here too
  - yodon 4y ago
    You seem to presuppose that the people involved in the team conversations about these metrics are stupid or clueless to a point of willfulness or intentionally bad actors. If that's the case, as with the case where you're somehow able to approve your own CR's, you face problems unrelated to the numbers. If those are the kind of problems you face, you need to fix them well before you try to bring in any processes designed to help a well functioning organization function better, including the case you called out where devs are approving code reviews without understanding them.

feoren 4y ago

> - Avg time to complete code review Doesn't this depend heavily on the size of each PR? Sometimes it makes sense to have small changes, sometimes to have sweeping refactors. These would have vastly different times to complete a code review, for reasons unrelated to team productivity.

yodon 4y ago
> Doesn't [Code Review time] depend heavily on the size of each PR? The amount of active "eyeballs on code" time to complete a code review can of course vary tremendously. That said, when I see code reviews taking a week to complete, only very very very rarely is that a sign that developers are routinely spending a solid week of eyeballs-on-code time completing those code reviews. Most of the time it involves a relatively small amount of focused review time and a huge amount of not actually getting around to starting doing the code review time. If everything in your shop is so well handled that you're not wasting any time waiting for code to get reviewed, in practice your shop is probably also already making good choices about how to structure code for efficient reviews as well (but that's more an anecdotal observation about correlations, not directly an observation about causality).
kqr 4y ago
Have you measured this in your organisation? In my experience, every team has a fairly distinct cut-off point where if you make PRs larger than that, they start to hang around in someone's review queue forever. Every time I've made a larger PR I've regretted it and subsequently split it up into smaller ones. I get higher quality feedback and faster feedback to boot that way.
- dkersten 4y ago
  > if you make PRs larger than that, they start to hang around in someone's review queue forever. Or you just get a "ship it" without more than a cursory glance.
mdoms 4y ago
Large PRs are one of the most serious anti patterns in day to day work.

puchatek 4y ago

Just the fact that these are team-oriented and not individual-oriented makes them above average valuable already

marcosdumay 4y ago

> Fraction of tasks started that never reach a customer If you optimize this down, you punish any kind of innovative or ambitious task.

lukeramsden 4y ago
Depends how you measure "reaching", i.e. all refactorings and optimisations all reach the customer technically "reach" the customer as soon as they're deployed (essentially, anything that meets its definition-of-done is "delivered). I think the wider idea is to measure work being interrupted and abandoned, or being abandoned due to not being necessary, both of which could be dubbed signs of inefficiency.
- marcosdumay 4y ago
  Hum, no. I'm talking exactly about abandoned work. If you don't abandon work, you either don't take any risky task, or you create a really shitty environment and will lose effectiveness soon because of it anyway. And yes, for some software you don't actually need to take any risky task. At least for a while. But if you are in a position to set that policy, you are very likely not in a position to have a clear enough view to know if your software is such one.
  - yodon 4y ago
    In practice I generally see the abandoned work fraction dominated by much less interesting and much less valuable poor planning decisions, false starts, and misalignment between teams. Genuine high-value risk taking is typically already so rare in most organizations that it doesn't even show up significantly in a count like this. Equally importantly, when those genuine risk taking activities are terminated they typically happen in the form of an experiment that reached a customer at some level as part of the decision to terminate the effort. If it reached a customer in some manner like that it's not in the bucket of tasks started that never reached a customer.
  - jrockway 4y ago
    This is worth a read: https://apenwarr.ca/log/20171213 https://apenwarr.ca/log/20171213 Specifically, look at the "PM can't make up their mind" graphs (https://apenwarr.ca/log/20171213#slide13 https://apenwarr.ca/log/20171213#slide13). That's the state that you're trying to avoid. I think it's fine to have tasks like "investigate the feasibility of X" and finding it infeasible. To me, that outcome is similar to delivering a feature (your customers will pay less for that than an actual feature, though, probably). What you want to avoid is "do X at any cost", and then the first steps are "integrate with third party service Y" "get a support contract for service Y" "do a giant refactoring to better use service Y", and then finding that X is actually infeasible. Now you have complexity, cost, and wasted months without having anything to give to the user. (Conflict of interest note: go ahead and do that. More work that never ships = more developers needed = higher pay for developers!)

splittingTimes 4y ago

They sure are nice, but how do you collect that data in an automated fashion?

rsanek 4y ago
Top two should be available through an ETL of whatever VCS is used. Next two depend on how fine-grained you want to be--either you don't automate because you stay high-level on the order of products, or you go lower-level and use feature flags or even basic logging for when a given feature is used. Not sure how you easily measure the last one though.
kqr 4y ago
Obvious follow-up: why do you need to automate it? You get decent precision by sampling just 10 % of what's going on. Depending on team size, that's manageable by just one person spending half an hour on it here and there. And in the process of asking questions and poking around this person learns a lot about what's happening.