9 ms·

Measuring an engineering organization

141 points by theptip 4y ago

ed_elliott_asc 4y ago
My only contribution to this is that I strongly believe you should only be measured on a system you built from scratch - inheriting someone else’s system and then you being measured on it is unfair.
- bcyn 4y ago
  Why do you believe that? If the goal were to measure your individual programming skills in a vacuum, then your recommendation makes sense. However, software engineering in a larger organization has a different goal -- it's team sport of building & operating software to serve the business purposes of that organization. With this goal in mind, one's ability to build upon & maintain an inherited codebase is definitely relevant to success.
mm686 4y ago
Here's a related article from one of the general partners at Coatue. The article covers a set of metrics, including DORA and SPACE, as well as productivity accelerators (and drags) that can help companies as they scale up. https://www.notion.so/coatue-external/Scaling-Up-Engineering-Productivity-1aab402d5b70437a83d5fb60f5398771 https://www.notion.so/coatue-external/Scaling-Up-Engineering...
lifeisstillgood 4y ago
What should we measure to measure productivity of journalists, of novel writers, of pop song composers? How do you measure the output of a engineer in a clothing factory or a short run machine tool shop? "Management" needs to stop thinking that software is an assembly line and that they are managing "workers". The actual workers are CPUs and GPUs - and coders "manage" those - coders are the new managers. The bullshit jobs people worry about are those managers who think they are arranging and orchestrating coders and if they shuffle them faster they will get better results. As more and more of the real world becomes eaten by software, measure that. I like the comparison to a engineer in a factory. In 1960 how did we measure the impact of an Detroit factory engineer? "Did the time to turn around the factory line drop this quarter?" That is a high level, impactful metric that has many inputs and would be a "managerial or executive level" metric with bonus attached Today that same metric works for "has the turnaround time for a new release of the data pipeline improved this quarter" Those are the metrics that will matter. Measure those.
- kovezd 4y ago
  > What should we measure to measure productivity of journalists? Visits, clicks, read time. > Novel writers? Sales, reviews. > Pop song composers? Hits. > "Management" needs to stop thinking that software is an assembly line and that they are managing "workers" I can't avoid to find funny the thought of telling management, as an engineer, what they should do. --- Good software engineers should embrace good productivity metrics (like the ones presented) as a meritocratic path to success. I find there are 2 distinct set of metrics for engineers: ability to delivery, and to create value. Both important, both difficult to measure, both possible to approximate, and both benefit the business.
  - hcarvalhoalves 4y ago
    > > What should we measure to measure productivity of journalists? > Visits, clicks, read time. > > Novel writers? > Sales, reviews. > > Pop song composers? > Hits. These are all proxies for revenue. So, you could just measure revenue. Then you fall into the trap of producing mediocre creative work to try to maximize revenue.
    - quanticle 4y ago
      If you're coding for someone else, rather than coding for yourself, why is this a trap? It might be satisfying to deliver a hyper-optimized end product, using optimal algorithms everywhere, with everything tuned so that it delivers the maximum performance with the minimum resource usage. But is that what your customer is paying for? Or are they paying for a website that works well, and is operational at the end of the month? Not everything has to be a work of high art. Pop art is fine too.
      - simondw 4y ago
        For one thing, total revenue over some longer time period is probably the actual thing to maximize. Short-term revenue may not be the best predictor of that, but we still need short-term metrics to evaluate performance.
  - 4y ago
hitekker 4y ago
Another day, another management article with a lot of opinions and no research. I find https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy and https://en.wikipedia.org/wiki/Surrogation https://en.wikipedia.org/wiki/Surrogation to be better starting points on the dangers of organizational measurement. > The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide. > Managers tend to use measures as surrogates for strategy, acting as if measures were in fact the strategy when making optimization decisions. This appears to occur even if a measure-maximizing choice ultimately works against the strategy. For those interested in history, there's a lot to learn about metric abuse from the Vietnam and Iraq Wars. For example: > On October 27, 1967, The Wall Street Journal ran an un-bylined blurb from its Washington, D.C., bureau on the front page talking about a “victory index.”: U.S. strategists seek a “victory index” to measure progress in the Vietnam War. They want a single statistic reflecting enemy infiltration, casualties, recruiting, hamlets pacified, roads cleared. Top U.S. intelligence officials, in a recent secret huddle, couldn’t work out an index; they get orders to keep trying. (from: https://www.theatlantic.com/technology/archive/2017/10/the-computer-that-predicted-the-us-would-win-the-vietnam-war/542046/ https://www.theatlantic.com/technology/archive/2017/10/the-c...)
- deleted 4y ago
  [deleted]
rukuu001 4y ago
It’s a good article. Key points for me: - measure teams not individuals - stick to measurements related to planning and operations - implement one measurement at a time I didn’t see it in the article (yes I skimmed) but I’d add that it’s beneficial to develop measurements in conjunction with engineering teams.
snidane 4y ago
You need to have a vision, then measure time of being blocked. If you're blocked, you're not progressing towards the vision. If you have a team of sufficiently competent people, you should expect to achieve your vision eventually. Problem of many organizations is they are progressing without a vision and have no idea when progress gets blocked. Since nobody really measures blockage, workers typically drop the thing that got blocked and go chase metrics on something else and don't tell anyone.
pdyc 4y ago
this topic is so close to my heart just yesterday i posted about my new project which measures these metrics(don't worry will not plug it here!) My take is that your measurment should give you some actionable insight and you should not measure absolute values. For example incident count, this looks like a useless metric to me because i can't tell what does high or low value mean, i also would need to look at team size to get anything useful out of it and that would not be very useful either. Same thing goes for lines of code, would 4 lines of lisp code mean i am less productive than 20 lines of c code? Instead measure ratio of outgoing requests to incoming request if this ratio is greater than one or approaching one than you will not be able to execute with your current plan since you are not able to dispose issues in time. Your team is in danger zone and there is a risk of burnout of your team and non delivery of the project. This is also useful because you can go to the management team and tell them you need additional headcount and present this data and you are likely to get your request approved since it is backed by data. Same thing goes for other metrics as well. Your metrics should be related to your goals they should clearly tell you if you can achieve your goal and how near/far you are from it. Last but not the least don't forget about human touch since you are managing humans. Metrics should just be the starting point to dig deeper to solve the organizational issue they should not be used as final goal.
_ewbb 4y ago
What does good team productivity look like? ROI? What kinds of metrics are used for planning and operations? Productivity sinks? I view engineering teams as a collection of highly driven professionals whos productivity levels are by default at 80% max daily. Their productivity is impacted by bad planning, bad developer experience, bad documentation, meetings without agendas, 1:1s with no support or actionable feedback, individual recognition instead of team recognition, etc. It's the engineering manager's job to remove the things that impact team productivity, measure the impact, report the issues affecting productivity, and plan to make them not happen again in the future. These are external to the team, things soaking up their time and making them stressed. When you try to measure an individuals productivity of any role, you are now impacting productivity and should be removed to allow them to do their best work. Use surveys to gather feedback from teams and allow teams to weed out, or help to reduce, internal productivity blockers. And please communicate this to all members of a team, it's one of the primary internal to the team stress sources I've experienced; role != authority Roles are a collection of responsibilities, they do not grant additional authority. From my experience what I've mentioned above would certainly make my day to day alot less stressful.
geekjock 4y ago
"What should we measure to improve developer productivity?" is a decades-old problem for leaders with no clear solution. There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach. Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity. A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com http://paper.getdx.com
- rqtwteye 4y ago
  “the things that slow them down or frustrate them” Whenever management makes a new attempt at getting more precise estimates I always tell them the only way to get things done is by doing them. So the number one goal should be to remove obstacles, simplify processes and make sure tools work to our benefit. For some reason nothing much ever changes and the question for better estimates returns. It’s pretty weird.
  - quickthrower2 4y ago
    Anything that is routinely easy to estimate can probably be automated away. And so will your competition.
- Cardinal7167 4y ago
  > "What should we measure to improve developer productivity?" How about nothing? Honest question. I am sick and tired of being treated like cattle. I've never seen a single convincing argument that any metric of my outputs has mapped to some productivity metric in a meaningful way, but I have absolutely seen them weaponized against me and others to punish and fire engineers. All of these metrics have only ever boiled down to a negative reinforcement mechanism, in my experience.
  - yamtaddle 4y ago
PragmaticPulp 4y ago
Measuring gets a bad rap, but proper measuring can be very valuable to engineers as well. If the company is only trying to measure developer productivity, they're missing the big picture. Developers will only get blamed when the number doesn't match some hidden expectation. Good organizations measure a lot of things, such as tech support effort (mentioned in this article), roadmap churn, number of unexpected feature demands that derail forecasts, and other metrics that evaluate the company rather than laying blame on a single group. Measuring roadmap churn and the impact of unexpected feature requests has been very helpful for me at metrics-heavy companies. When management starts asking why things haven't been shipped yet, it's amazing to pull out a quantified list of all of their unplanned requests, roadmap changes, and other management issues that set everyone back.
- e_i_pi_2 4y ago
  I think a lot of the bad rap comes from how the measurements are used - I personally like to know my trends for how many hours/commits/etc I did in different areas during a time period. I use Wakatime[1] in my editor and love checking in every once in a while. However I would never ask everyone on my team to install it and share the dashboards, because then the measurements may be used the wrong way, e.g. "Alice did way more commits than Bob last week, so that's bad for Bob". As another example I like using story points/velocity to get a feel for the team, but I don't think you can extrapolate on that data and expect the results to be valid. I think most of the frustration from devs come from the measurements being used to try to plan out the future when that isn't really possible. IMO you can choose a set of features OR a release date but definitely not both. I'm no expert but I've read a fair amount on different strategies to try and make timelines for software and my big take-away at this point is "don't make timelines for software. If you _have_ to, then keep it probabalistic and don't promise anything by an exact date". I think we as devs need to kinda educate the rest of the business that it's a tradeoff between features and release dates. I like the "iron triangle" model but you can't just add more people to a team and expect things to be done faster (Mythical Man-Month), so it's really just scope VS release date. [1]: https://wakatime.com https://wakatime.com
- bcyn 4y ago
  Could you share some specific tactics/metrics you've defined to measure things like roadmap churn or unexpected feature demands? This problem definitely resonates with me, and I have some very primitive ideas of tracking (i.e. tagging tickets that fall into these buckets), but I would love to learn from your experience if you're willing to share!
splittingTimes 4y ago
I agree with most of the sentiments in the article and am always keen on learning new meaningful, i.e. not individual but high-level measures. The notion of "measuring something is better than nothing" is not helpful, I feel. You not measure what is just easy to measure, but what is actually imporant and gives real business insight. This, as the article states, is typically very hard to define and execute. Also, I would widen the perspective of "CTO needs to provide data to CEO" / engineering effectivness a bit, as this is not the most important thing you need to know. Ultimately, as a company you wanna know if your business is going in the right direction or not. You want to spend more money on things & activities that add to the value of your product offering in the eyes of a customer so that the customer will pay for it and you want to spend less money on things customers will not pay for. But what are the value-adding activities and which are non-value-adding? When will a customer buy your product or service? For me, this sets a certain order of importance of what things to measure/quantify to answer the following questions: 1. Deliver customer value. Are we building the right product? 2. Generate business value. Can we generate revenue? 3. High value product. Do we keep quality high and build the right next features? 4. Code quality. Are we building the product right so it is maintainable and exensible? 5. Team chemistry. Is the team aligned on the goal and healthy in their interactions and spirit? 6. Process efficiency. Is success repeatable? === (1) Answering that should be not too hard as you can measure any kind of customer feedback on your products, be it youtube likes, alpha tester feedback, support calls, surveys. (2) Once you know you have a product customers want and like, you need to know is the customer facing part of your organization (marketing, sales, training & education and support) connecting to said customer, so that they can sell and actually generate revenue? This is much harder to get data for. You can measure the number of licenses, lost and new customers, or the trends in the business volume of services, but what does that tell you about the ability of your organization to be able to sell a good product to customers? I am not sure here. (3) is to reflect on the business success (ROI) on new features that you implement. Do you keep building a high value product? Are we effective in communicating the new features value proposition to our customers? There is soo much to measure here. Feature level: Measure via BI the business impact of new features, how often are they used? Measure the resources needed to deliver a feature. Measure number of support cases / bugs reported per feature. Measure the estimated time vs the taken time for a feature. Quality of the overall product: Measure the yearly mean number of support cases per week. Measure the yearly mean number of bug reports per week. Measure the yearly mean number of crash reports per week. Measure number of major field incidences per year. Do we make the customer feel cared about after the sale? Measure lead time to resolve support calls. Measure customer ratings of support calls. (4) Code quality like compiler/sonar warnings or test coverage are the easy measures, but might not give you insight. I prefer to look more highlevel again at the product quality which ties into (3), like How often do tickets come back from testing to development? Measure number of regressions reported by customers after a release. Measure number of (real) hotfixes needed after a release. Answering the more fundamental questions of code quality like "how easy is it to add new features" is a bit more difficult. I have not found good measures yet. For (5) totally agree with the article, you should never measure how many lines of code were produced, how many tickets closed or the like. The question of team happiness is most important for team work. Ther are good tools for that like OfficeVibe or Glint. Measure the employee turnover. Measure the number of uninterrupted hours of work time / total time present at work per week. Measure the mean Office Vibe score. Many further answers could be drawn off of Office vibe: Are people at ease, having a good time and enjoying interactions with their peers? Is there no sense that single individuals try to succeed in spite of the efforts of those around them? Was the work a joint product? Was everybody proud of its quality? Do they take enjoyment in their work? Is there trust and mutual esteem among the peers? For (6) you want to know: Are our processes such that success are repeatable? On a company level: Measure number of botched releases / roll backs. Measure number of failed audits per year. Measure number of open CAPAs per year. Per Team level: How long are compile/ CICD times? How long do code reviews lie around before taken up? How easy is on-boarding of new employees? etc

141 points by theptip 4y ago

My only contribution to this is that I strongly believe you should only be measured on a system you built from scratch - inheriting someone else’s system and then you being measured on it is unfair.

bcyn 4y ago
Why do you believe that? If the goal were to measure your individual programming skills in a vacuum, then your recommendation makes sense. However, software engineering in a larger organization has a different goal -- it's team sport of building & operating software to serve the business purposes of that organization. With this goal in mind, one's ability to build upon & maintain an inherited codebase is definitely relevant to success.

mm686 4y ago

Here's a related article from one of the general partners at Coatue. The article covers a set of metrics, including DORA and SPACE, as well as productivity accelerators (and drags) that can help companies as they scale up. https://www.notion.so/coatue-external/Scaling-Up-Engineering-Productivity-1aab402d5b70437a83d5fb60f5398771 https://www.notion.so/coatue-external/Scaling-Up-Engineering...

lifeisstillgood 4y ago

What should we measure to measure productivity of journalists, of novel writers, of pop song composers? How do you measure the output of a engineer in a clothing factory or a short run machine tool shop? "Management" needs to stop thinking that software is an assembly line and that they are managing "workers". The actual workers are CPUs and GPUs - and coders "manage" those - coders are the new managers. The bullshit jobs people worry about are those managers who think they are arranging and orchestrating coders and if they shuffle them faster they will get better results. As more and more of the real world becomes eaten by software, measure that. I like the comparison to a engineer in a factory. In 1960 how did we measure the impact of an Detroit factory engineer? "Did the time to turn around the factory line drop this quarter?" That is a high level, impactful metric that has many inputs and would be a "managerial or executive level" metric with bonus attached Today that same metric works for "has the turnaround time for a new release of the data pipeline improved this quarter" Those are the metrics that will matter. Measure those.

kovezd 4y ago
> What should we measure to measure productivity of journalists? Visits, clicks, read time. > Novel writers? Sales, reviews. > Pop song composers? Hits. > "Management" needs to stop thinking that software is an assembly line and that they are managing "workers" I can't avoid to find funny the thought of telling management, as an engineer, what they should do. --- Good software engineers should embrace good productivity metrics (like the ones presented) as a meritocratic path to success. I find there are 2 distinct set of metrics for engineers: ability to delivery, and to create value. Both important, both difficult to measure, both possible to approximate, and both benefit the business.
- hcarvalhoalves 4y ago
  > > What should we measure to measure productivity of journalists? > Visits, clicks, read time. > > Novel writers? > Sales, reviews. > > Pop song composers? > Hits. These are all proxies for revenue. So, you could just measure revenue. Then you fall into the trap of producing mediocre creative work to try to maximize revenue.
  - quanticle 4y ago
    If you're coding for someone else, rather than coding for yourself, why is this a trap? It might be satisfying to deliver a hyper-optimized end product, using optimal algorithms everywhere, with everything tuned so that it delivers the maximum performance with the minimum resource usage. But is that what your customer is paying for? Or are they paying for a website that works well, and is operational at the end of the month? Not everything has to be a work of high art. Pop art is fine too.
    - simondw 4y ago
      For one thing, total revenue over some longer time period is probably the actual thing to maximize. Short-term revenue may not be the best predictor of that, but we still need short-term metrics to evaluate performance.
- 4y ago

hitekker 4y ago

Another day, another management article with a lot of opinions and no research. I find https://en.wikipedia.org/wiki/McNamara_fallacy https://en.wikipedia.org/wiki/McNamara_fallacy and https://en.wikipedia.org/wiki/Surrogation https://en.wikipedia.org/wiki/Surrogation to be better starting points on the dangers of organizational measurement. > The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide. > Managers tend to use measures as surrogates for strategy, acting as if measures were in fact the strategy when making optimization decisions. This appears to occur even if a measure-maximizing choice ultimately works against the strategy. For those interested in history, there's a lot to learn about metric abuse from the Vietnam and Iraq Wars. For example: > On October 27, 1967, The Wall Street Journal ran an un-bylined blurb from its Washington, D.C., bureau on the front page talking about a “victory index.”: U.S. strategists seek a “victory index” to measure progress in the Vietnam War. They want a single statistic reflecting enemy infiltration, casualties, recruiting, hamlets pacified, roads cleared. Top U.S. intelligence officials, in a recent secret huddle, couldn’t work out an index; they get orders to keep trying. (from: https://www.theatlantic.com/technology/archive/2017/10/the-computer-that-predicted-the-us-would-win-the-vietnam-war/542046/ https://www.theatlantic.com/technology/archive/2017/10/the-c...)

deleted 4y ago
[deleted]

rukuu001 4y ago

It’s a good article. Key points for me: - measure teams not individuals - stick to measurements related to planning and operations - implement one measurement at a time I didn’t see it in the article (yes I skimmed) but I’d add that it’s beneficial to develop measurements in conjunction with engineering teams.

snidane 4y ago

You need to have a vision, then measure time of being blocked. If you're blocked, you're not progressing towards the vision. If you have a team of sufficiently competent people, you should expect to achieve your vision eventually. Problem of many organizations is they are progressing without a vision and have no idea when progress gets blocked. Since nobody really measures blockage, workers typically drop the thing that got blocked and go chase metrics on something else and don't tell anyone.

pdyc 4y ago

this topic is so close to my heart just yesterday i posted about my new project which measures these metrics(don't worry will not plug it here!) My take is that your measurment should give you some actionable insight and you should not measure absolute values. For example incident count, this looks like a useless metric to me because i can't tell what does high or low value mean, i also would need to look at team size to get anything useful out of it and that would not be very useful either. Same thing goes for lines of code, would 4 lines of lisp code mean i am less productive than 20 lines of c code? Instead measure ratio of outgoing requests to incoming request if this ratio is greater than one or approaching one than you will not be able to execute with your current plan since you are not able to dispose issues in time. Your team is in danger zone and there is a risk of burnout of your team and non delivery of the project. This is also useful because you can go to the management team and tell them you need additional headcount and present this data and you are likely to get your request approved since it is backed by data. Same thing goes for other metrics as well. Your metrics should be related to your goals they should clearly tell you if you can achieve your goal and how near/far you are from it. Last but not the least don't forget about human touch since you are managing humans. Metrics should just be the starting point to dig deeper to solve the organizational issue they should not be used as final goal.

_ewbb 4y ago

What does good team productivity look like? ROI? What kinds of metrics are used for planning and operations? Productivity sinks? I view engineering teams as a collection of highly driven professionals whos productivity levels are by default at 80% max daily. Their productivity is impacted by bad planning, bad developer experience, bad documentation, meetings without agendas, 1:1s with no support or actionable feedback, individual recognition instead of team recognition, etc. It's the engineering manager's job to remove the things that impact team productivity, measure the impact, report the issues affecting productivity, and plan to make them not happen again in the future. These are external to the team, things soaking up their time and making them stressed. When you try to measure an individuals productivity of any role, you are now impacting productivity and should be removed to allow them to do their best work. Use surveys to gather feedback from teams and allow teams to weed out, or help to reduce, internal productivity blockers. And please communicate this to all members of a team, it's one of the primary internal to the team stress sources I've experienced; role != authority Roles are a collection of responsibilities, they do not grant additional authority. From my experience what I've mentioned above would certainly make my day to day alot less stressful.

geekjock 4y ago

"What should we measure to improve developer productivity?" is a decades-old problem for leaders with no clear solution. There finally seems to be some level of consensus that output metrics like lines of code, # of PRs, and commits, are an ineffective approach. Lean metrics like cycle time and lead time can be a helpful high-level diagnostic, but they're far from an indicator of effectiveness or productivity. A new approach being adopted by many organizations is to focus on the actual experiences of developers... the things that slow them down or frustrate them... and turn these into measurements that guide improvement. I'm the founder of getdx.com where we're publishing research on this: http://paper.getdx.com http://paper.getdx.com

rqtwteye 4y ago
“the things that slow them down or frustrate them” Whenever management makes a new attempt at getting more precise estimates I always tell them the only way to get things done is by doing them. So the number one goal should be to remove obstacles, simplify processes and make sure tools work to our benefit. For some reason nothing much ever changes and the question for better estimates returns. It’s pretty weird.
- quickthrower2 4y ago
  Anything that is routinely easy to estimate can probably be automated away. And so will your competition.
Cardinal7167 4y ago
> "What should we measure to improve developer productivity?" How about nothing? Honest question. I am sick and tired of being treated like cattle. I've never seen a single convincing argument that any metric of my outputs has mapped to some productivity metric in a meaningful way, but I have absolutely seen them weaponized against me and others to punish and fire engineers. All of these metrics have only ever boiled down to a negative reinforcement mechanism, in my experience.
- yamtaddle 4y ago

PragmaticPulp 4y ago

Measuring gets a bad rap, but proper measuring can be very valuable to engineers as well. If the company is only trying to measure developer productivity, they're missing the big picture. Developers will only get blamed when the number doesn't match some hidden expectation. Good organizations measure a lot of things, such as tech support effort (mentioned in this article), roadmap churn, number of unexpected feature demands that derail forecasts, and other metrics that evaluate the company rather than laying blame on a single group. Measuring roadmap churn and the impact of unexpected feature requests has been very helpful for me at metrics-heavy companies. When management starts asking why things haven't been shipped yet, it's amazing to pull out a quantified list of all of their unplanned requests, roadmap changes, and other management issues that set everyone back.

e_i_pi_2 4y ago
I think a lot of the bad rap comes from how the measurements are used - I personally like to know my trends for how many hours/commits/etc I did in different areas during a time period. I use Wakatime[1] in my editor and love checking in every once in a while. However I would never ask everyone on my team to install it and share the dashboards, because then the measurements may be used the wrong way, e.g. "Alice did way more commits than Bob last week, so that's bad for Bob". As another example I like using story points/velocity to get a feel for the team, but I don't think you can extrapolate on that data and expect the results to be valid. I think most of the frustration from devs come from the measurements being used to try to plan out the future when that isn't really possible. IMO you can choose a set of features OR a release date but definitely not both. I'm no expert but I've read a fair amount on different strategies to try and make timelines for software and my big take-away at this point is "don't make timelines for software. If you _have_ to, then keep it probabalistic and don't promise anything by an exact date". I think we as devs need to kinda educate the rest of the business that it's a tradeoff between features and release dates. I like the "iron triangle" model but you can't just add more people to a team and expect things to be done faster (Mythical Man-Month), so it's really just scope VS release date. [1]: https://wakatime.com https://wakatime.com
bcyn 4y ago
Could you share some specific tactics/metrics you've defined to measure things like roadmap churn or unexpected feature demands? This problem definitely resonates with me, and I have some very primitive ideas of tracking (i.e. tagging tickets that fall into these buckets), but I would love to learn from your experience if you're willing to share!

splittingTimes 4y ago

I agree with most of the sentiments in the article and am always keen on learning new meaningful, i.e. not individual but high-level measures. The notion of "measuring something is better than nothing" is not helpful, I feel. You not measure what is just easy to measure, but what is actually imporant and gives real business insight. This, as the article states, is typically very hard to define and execute. Also, I would widen the perspective of "CTO needs to provide data to CEO" / engineering effectivness a bit, as this is not the most important thing you need to know. Ultimately, as a company you wanna know if your business is going in the right direction or not. You want to spend more money on things & activities that add to the value of your product offering in the eyes of a customer so that the customer will pay for it and you want to spend less money on things customers will not pay for. But what are the value-adding activities and which are non-value-adding? When will a customer buy your product or service? For me, this sets a certain order of importance of what things to measure/quantify to answer the following questions: 1. Deliver customer value. Are we building the right product? 2. Generate business value. Can we generate revenue? 3. High value product. Do we keep quality high and build the right next features? 4. Code quality. Are we building the product right so it is maintainable and exensible? 5. Team chemistry. Is the team aligned on the goal and healthy in their interactions and spirit? 6. Process efficiency. Is success repeatable? === (1) Answering that should be not too hard as you can measure any kind of customer feedback on your products, be it youtube likes, alpha tester feedback, support calls, surveys. (2) Once you know you have a product customers want and like, you need to know is the customer facing part of your organization (marketing, sales, training & education and support) connecting to said customer, so that they can sell and actually generate revenue? This is much harder to get data for. You can measure the number of licenses, lost and new customers, or the trends in the business volume of services, but what does that tell you about the ability of your organization to be able to sell a good product to customers? I am not sure here. (3) is to reflect on the business success (ROI) on new features that you implement. Do you keep building a high value product? Are we effective in communicating the new features value proposition to our customers? There is soo much to measure here. Feature level: Measure via BI the business impact of new features, how often are they used? Measure the resources needed to deliver a feature. Measure number of support cases / bugs reported per feature. Measure the estimated time vs the taken time for a feature. Quality of the overall product: Measure the yearly mean number of support cases per week. Measure the yearly mean number of bug reports per week. Measure the yearly mean number of crash reports per week. Measure number of major field incidences per year. Do we make the customer feel cared about after the sale? Measure lead time to resolve support calls. Measure customer ratings of support calls. (4) Code quality like compiler/sonar warnings or test coverage are the easy measures, but might not give you insight. I prefer to look more highlevel again at the product quality which ties into (3), like How often do tickets come back from testing to development? Measure number of regressions reported by customers after a release. Measure number of (real) hotfixes needed after a release. Answering the more fundamental questions of code quality like "how easy is it to add new features" is a bit more difficult. I have not found good measures yet. For (5) totally agree with the article, you should never measure how many lines of code were produced, how many tickets closed or the like. The question of team happiness is most important for team work. Ther are good tools for that like OfficeVibe or Glint. Measure the employee turnover. Measure the number of uninterrupted hours of work time / total time present at work per week. Measure the mean Office Vibe score. Many further answers could be drawn off of Office vibe: Are people at ease, having a good time and enjoying interactions with their peers? Is there no sense that single individuals try to succeed in spite of the efforts of those around them? Was the work a joint product? Was everybody proud of its quality? Do they take enjoyment in their work? Is there trust and mutual esteem among the peers? For (6) you want to know: Are our processes such that success are repeatable? On a company level: Measure number of botched releases / roll backs. Measure number of failed audits per year. Measure number of open CAPAs per year. Per Team level: How long are compile/ CICD times? How long do code reviews lie around before taken up? How easy is on-boarding of new employees? etc