4 ms·

I like seeing actual numbers about what Good is meant to be for this metrics - it's easy for a devlops advocate to highlight a metrics and push for further inve

I like seeing actual numbers about what Good is meant to be for this metrics - it's easy for a devlops advocate to highlight a metrics and push for further investment in it, despite it not being a major weakness of the organisation otherwise.

isthisit 4y ago
Author here. The targets you set for Delivery Lead Time and Deployment Frequency should be determined by your organisation and your team. Context matters. I've been at an organisation where upper management introduced the Accelerate/DORA targets verbatim. These numbers did not resonate at all with engineers and so saw little to no adoption. Accelerate was published four years ago and understanding of the 'four DORA metrics' has matured. Štěpán Davidovič wrote a good piece Incident Metrics in SRE [1] where he pulls apart the Mean Time to Recovery Metric. I've written an overview of the DORA metrics and what to use instead - SLOs & SLIs [2] [1] https://web.archive.org/web/20210917022148/https://static.googleusercontent.com/media/sre.google/en//static/pdf/incident_metrics_in_sre.pdf https://web.archive.org/web/20210917022148/https://static.go... [2] https://isthisit.nz/posts/2022/state-of-the-dora-devops-metrics/ https://isthisit.nz/posts/2022/state-of-the-dora-devops-metr...
numbsafari 4y ago
External benchmarking is a tricky thing, especially with a number like this, which is derived from a self-reported survey. The important thing is that those teams perceive they can quickly deploy a change. Perhaps they do, at times, but as the author of the OP points out, that’s not really revealed in the source literature, and being able to isn’t the same thing as often or frequently flexing that muscle. I would start with “can you even measure this”. Just getting something like this instrumented for your team is probably going to drive improvements. Then you can put it on a Shewhart/PDCA Cycle driven primarily by the team itself through its retrospective and prioritization process. “Has this metric slipped? Do we feel like there are blocks in the deploy pipeline? Are those blocks more important than other work right now? What changes does the team feel most invested in?” Fostering that conversation is really important. Grabbing these metrics, poorly measuring them, and then trying to hold a team accountable to some arbitrary, external, non-contextual measure is a sure way to kill morale and productivity, hinder future leadership efforts, and waste time and resources and non-priorities. Of course, doing nothing will accomplish the same. We do this kind of work with hospitals and their process metrics. Our measurements are always based on raw data, never self-reported metrics. Frequently, team’s self-reported perceptions prior to measurement is the most telling thing. When I worked in k-12 Ed, the same was true with teachers and their students: they perceived their students as great writers, but they were often much worse when measured quantitatively by normalized graders. What is important in both cases isn’t blindly pushing to improve those metrics, but developing an understanding of why that perceived difference exists, why the numbers are the way they are, and ultimately, whether and what kinds of differences you can make. In the case of hospitals, there are some units/teams where you don’t want high performance numbers, because it might lead to higher error rates. In Ed, there might be limits to what you can do as a single-subject teacher focused on a single year in a group of children’s lives. But you can understand each groups challenges and strengths a little better and maybe adapt accordingly. Blindly pushing top-line metrics is not the same as blindly doing the same thing with no measurement or learning. Both are bad.
- _hcuq 4y ago
  There is a general rule in social and economic life” wrote W. Brian Arthur, an economist at the Santa Fe Institute, “given any system, people will find a way to exploit it. Or to say this more succinctly: All systems will be gamed”. https://waldencroft.com/gaming-the-system-the-unintended-consequences-of-measuring-the-wrong-things/ https://waldencroft.com/gaming-the-system-the-unintended-con...
  - numbsafari 4y ago
    Which is why MBO is ultimately not a good management tool and is clearly not serving many organizations as well as they need, especially when applied to the management of Systems/Software Engineers who are all too aware of this fact. If you want to be successful, you have to focus on intrinsic motivations of these kinds of staff. By and large, they do not respond effectively to extrinsic motivations. They may respond "positively", but not necessarily "effectively". It's difficult, though, in a world where any non-quantifiable evaluation is rightly viewed with incredible suspicion. It forces managers and leaders to walk a hard line.

isthisit 4y ago

Author here. The targets you set for Delivery Lead Time and Deployment Frequency should be determined by your organisation and your team. Context matters. I've been at an organisation where upper management introduced the Accelerate/DORA targets verbatim. These numbers did not resonate at all with engineers and so saw little to no adoption. Accelerate was published four years ago and understanding of the 'four DORA metrics' has matured. Štěpán Davidovič wrote a good piece Incident Metrics in SRE [1] where he pulls apart the Mean Time to Recovery Metric. I've written an overview of the DORA metrics and what to use instead - SLOs & SLIs [2] [1] https://web.archive.org/web/20210917022148/https://static.googleusercontent.com/media/sre.google/en//static/pdf/incident_metrics_in_sre.pdf https://web.archive.org/web/20210917022148/https://static.go... [2] https://isthisit.nz/posts/2022/state-of-the-dora-devops-metrics/ https://isthisit.nz/posts/2022/state-of-the-dora-devops-metr...

numbsafari 4y ago

External benchmarking is a tricky thing, especially with a number like this, which is derived from a self-reported survey. The important thing is that those teams perceive they can quickly deploy a change. Perhaps they do, at times, but as the author of the OP points out, that’s not really revealed in the source literature, and being able to isn’t the same thing as often or frequently flexing that muscle. I would start with “can you even measure this”. Just getting something like this instrumented for your team is probably going to drive improvements. Then you can put it on a Shewhart/PDCA Cycle driven primarily by the team itself through its retrospective and prioritization process. “Has this metric slipped? Do we feel like there are blocks in the deploy pipeline? Are those blocks more important than other work right now? What changes does the team feel most invested in?” Fostering that conversation is really important. Grabbing these metrics, poorly measuring them, and then trying to hold a team accountable to some arbitrary, external, non-contextual measure is a sure way to kill morale and productivity, hinder future leadership efforts, and waste time and resources and non-priorities. Of course, doing nothing will accomplish the same. We do this kind of work with hospitals and their process metrics. Our measurements are always based on raw data, never self-reported metrics. Frequently, team’s self-reported perceptions prior to measurement is the most telling thing. When I worked in k-12 Ed, the same was true with teachers and their students: they perceived their students as great writers, but they were often much worse when measured quantitatively by normalized graders. What is important in both cases isn’t blindly pushing to improve those metrics, but developing an understanding of why that perceived difference exists, why the numbers are the way they are, and ultimately, whether and what kinds of differences you can make. In the case of hospitals, there are some units/teams where you don’t want high performance numbers, because it might lead to higher error rates. In Ed, there might be limits to what you can do as a single-subject teacher focused on a single year in a group of children’s lives. But you can understand each groups challenges and strengths a little better and maybe adapt accordingly. Blindly pushing top-line metrics is not the same as blindly doing the same thing with no measurement or learning. Both are bad.

_hcuq 4y ago
There is a general rule in social and economic life” wrote W. Brian Arthur, an economist at the Santa Fe Institute, “given any system, people will find a way to exploit it. Or to say this more succinctly: All systems will be gamed”. https://waldencroft.com/gaming-the-system-the-unintended-consequences-of-measuring-the-wrong-things/ https://waldencroft.com/gaming-the-system-the-unintended-con...
- numbsafari 4y ago
  Which is why MBO is ultimately not a good management tool and is clearly not serving many organizations as well as they need, especially when applied to the management of Systems/Software Engineers who are all too aware of this fact. If you want to be successful, you have to focus on intrinsic motivations of these kinds of staff. By and large, they do not respond effectively to extrinsic motivations. They may respond "positively", but not necessarily "effectively". It's difficult, though, in a world where any non-quantifiable evaluation is rightly viewed with incredible suspicion. It forces managers and leaders to walk a hard line.