6 ms·

When I switched from engineering to management, I spent a long time thinking about software estimates, read about many different approaches, and argued with man

When I switched from engineering to management, I spent a long time thinking about software estimates, read about many different approaches, and argued with many other managers (I’ll try to write up my own blog post for the next time the topic comes up on here). The framing in this blog post is better than some (it at least discusses uncertainty) but I believe it’s still basically the wrong framework. People (managers) like to assume that estimates are a random variable, and software projects are samples from project space. Estimates can be produced for a project by analogizing it to similar projects. This is not true at all. You cannot reason about software projects by analogy. Some assertions (that I will explore in said promised blog post): - Software development is chaotic: arbitrarily small changes in the input (project requirements, who’s implementing it, stakeholders, team, etc) may lead to arbitrarily large changes in time-to-completion (including “this project is now impossible”). In short, any project may explode at any moment prior to completion. - The risk of a project is particularly sensitive to the engineer implementing it, and depends on that person’s specific knowledge. Have they used these APIs or this database before? Any part of a project that the engineer hasn’t used before represents potentially unbounded technical risk. - the way to manage this risk is to include buffer in the project timeline, and, critically, to use that buffer not for development tasks that were harder than expected, but for re-planning and re-negotiating the deliverables with stakeholders, to un-explode the project. Agile builds this re-evaluation into the project lifecycle, which is the main (maybe only?) thing is does really right.

necovek 2y ago
That's too much of a carrot IMHO: I would have appreciated it more if you dove deeper into one of those points instead of promising a blog post on a dozen. Edit: thanks for editing the comment to make it clearer what are your points. This makes my comment somewhat useless :)
necovek 2y ago
I agree that the points you raise are important to realise and affect delivery duration significantly. I, however, disagree with that approach to managing delivery: if you tie in expected value with deliverables, you empower and enable engineering teams to smartly cut scope and deliver in shorter amount of time the biggest chunk of the value. It's usually only described as "cutting scope", but it requires real creativity and smarts on the engineering team (they can best tell what's feasible in remaining time, when they have tackled all the unknown unknowns you bring up) to do that in a way that still brings the most of the original value. If you move this to your "renegotiation" step, it becomes too slow and hurts delivery as well.
- msteffen 2y ago
  I basically agree (I think) in that I don’t think there needs to be a separate “cutting scope” phase—one should it as soon as they realize they need to. And I certainly agree that it’s incumbent on engineering to invent alternatives and propose them when we realize that we need to cut scope (for the reasons you mentioned). FWIW, though, engineers shouldn’t decide what to cut (i.e. choose alternatives) in a vacuum, in my experience. I’ve been in plenty of meetings with product/sales/support where they say “it’s better for the whole project to slip than to release it without this one detail” or “we would give up what seems like basic usability to get one particular piece of polish”
  - necovek 2y ago
    Yes, totally agreed that engineers should not "decide" in a vacuum. But if they understand what value was supposed to be brought, have a decent product/customer focus, and are creative enough, they can propose good alternatives and not stall the delivery. I've also experienced things you mention, but it was always in orgs where everything, including any small feature, was treated as a "big bet": unsupported by metrics (no matter how imperfect), but instead wishful thinking that it will bring meaningful improvement. As such, you can't come up with anything that's an equally good improvement with less effort because there is no baseline to compare against.
    - msteffen 2y ago
      > I've also experienced things you mention, but it was always in orgs where everything, including any small feature, was treated as a "big bet": unsupported by metrics (no matter how imperfect), but instead wishful thinking that it will bring meaningful improvement. As such, you can't come up with anything that's an equally good improvement with less effort because there is no baseline to compare against. Interesting, that’s exactly the situation I was in, but I never connected the lack of metrics to these kinds of requests. TIL. I feel like I have lot to say about how this manifested. Product direction was very heavily guided by existing customers (because support could say “we have these three customers asking for X”), somewhat guided by closing deals (because sales could say “we have a $$$ deal that the customer says will close if we deliver Y”) and hardly guided at all by the broader market, because product’s suggestions could only ever be supported by speculation and vibes. But we were B2B, so I don’t even know what good metrics would’ve looked like—it’s not like we had billions of users
rixed 2y ago
I find your framing interesting. But most importantly, even if nothing changes in the requirements or the team, the mere new information that is learned while implementing the project is often enough to change the estimate considerably.
- msteffen 2y ago
  Yes! In my last role, I aggressively pushed the concept of “technical risk” to try and get people used to this idea. Sometimes the computers don’t work the way you thought they did at the beginning. This is a normal and ubiquitous form of risk that just needs to be managed like any other type of risk, with prototypes, multiple revisions of the implementation (each introducing additional risk), occasionally re-scoping, etc. People outside engineering sometimes don’t like it because the risk is human in origin—it comes from engineers not knowing things, and our job is to know things—but until there’s an omniscient engineer, this risk will continue to exist.
  - rixed 2y ago
    Yes, this fits my experience as well. Oftentimes I've heard people who are not and have never been software engineers ask "When we want to built a bridge, engineers can tell how long it will take and how expensive it will be, how come with software engineers it's never the case?" (often implied: you software engineers have it easy, slackers! -- I believe we put this blame on ourselves with our immature culture, but that's another discussion) What I usually answer to that is that, to begin with, that if they had actually dealed with big construction projects like bridges then they would not idealize those so much. And secondly, that this is not the right analogy. Bridge construction is a much stabler science, it had no "complexity explosion". A better analogy would be geophysical prospection : the theory it's based on is sound and mature but the unknowns dominate everything in predicting the outcome.
    - msteffen 2y ago
      If you haven’t seen it (and need something to show the “what about bridges” crew), Hillel Wayne’s “crossover project” series of posts[^1] on this is brilliant. I like to show people this quote from it: > One person talked about how frustrating it is to start work on a bridge foundation, only to find that this particular soil freezes in a weird way that makes it liquefy too much in an earthquake. Back to the drawing board. [^1]: https://www.hillelwayne.com/post/we-are-not-special/ https://www.hillelwayne.com/post/we-are-not-special/
nine_zeros 2y ago
- the way to manage this risk is to include buffer in the project timeline, and, critically, to use that buffer not for development tasks that were harder than expected, but for re-planning and re-negotiating the deliverables with stakeholders, to un-explode the project. Agile builds this re-evaluation into the project lifecycle, which is the main (maybe only?) thing is does really right. This is the only right answer. Unfortunately, engineering leadership is too detached from understanding how software works. They think of projects like contractor painting jobs. So easy to estimate - sq. ft * number of people * number of hours. But software is not like that. Software is constantly changing every hour, every day. As a result, any estimation is changing every hour every day. The best thing is for leadership to start acknowledging the realm they are dealing with. If they can't they should step down. But realistically, engineers should always give 400% padding with estimates. The root cause of this estimation is poor leadership. It is not engineering team's problem that detached management doesn't understand software.
wkirby 2y ago
It's been several years since my colleague put this post together, but I think for us it's proven to be a very constructive framework for talking to clients. It's probably worth calling out that we are, specifically, outside contractors providing estimates for clients --- usually clients at the start of their software development journey. I think a lot of our framework holds for developers working on in-house engineering teams, but there are bound to be some differences. That said, I'll be curious to read your post whenever it goes up. Namely because I don't know what the conclusion of your comment is. Whether it's as a vendor or as an engineer on staff, estimates are a hard requirement of software development. Most stakeholders are not developers, and you can only educate the recipients of your estimates so much on the nuances of what is and isn't hard to accommodate. I don't really disagree with any of your assertions --- but isn't the re-evaluation process simply... more estimation? Aren't accounting for project risks --- like which developer is going to pick up the task, what requirements are likely or unlikely to change --- part of producing a quality estimate? Communicating expectations is hard, and to me, one of the defining lines between junior and senior developers is the ability to clearly account for expected risks, and identify plausible but unlikely risks, and to incorporate mitigation into the plan of attack.
- msteffen 2y ago
  I agree, and in general I am very pro-estimates (they switch your team from cooperative scheduling to preemptive scheduling, and they’re an important tool for managing the chaos of development), but the statement “we expect this project to take eight weeks, but it could take 24 weeks in the worst case” is one I would be very loathe to make. Granted, I’m coming from a B2B startup rather than a consultancy, so I was working with a mixture of internal (product, sales, support) and external (customers) stakeholders. The perspective I would try to give people, though, was this: If we discover a bomb partway through the project (oops, groups are limited to 100 members in this auth system, so we will need to find a new system and migrate or else not support groups) then 8 weeks will no longer be enough, but maybe neither is 24. The right thing for us to do in that situation, IMO, is go back to the stakeholders and align on a new plan. Maybe groups will come in v2, or some groups will work and some won’t, or maybe we do the migration and add it to the estimate, or whatever. One shouldn’t use the 16 buffer weeks between 8 and 24 to sneak in a backend migration (which itself might be a 16-week project, or might not), but to pause and re-align on a plan everyone likes. When I was giving estimates, I would try to frame them as “we’ll spend eight weeks trying to accomplish this list of things. If we discover an issue that prevents us from finishing the list for any reason, we’ll come back to you as early as possible with some alternative proposals and re-assess”. Sometimes people felt that this meant we just weren’t willing to commit to our estimates, and this idea of “software development is chaotic” is what I would say, to explain that the issue wasn’t a lack of motivation, but an inescapable knowledge risk that needed to be managed.

necovek 2y ago

That's too much of a carrot IMHO: I would have appreciated it more if you dove deeper into one of those points instead of promising a blog post on a dozen. Edit: thanks for editing the comment to make it clearer what are your points. This makes my comment somewhat useless :)

necovek 2y ago

I agree that the points you raise are important to realise and affect delivery duration significantly. I, however, disagree with that approach to managing delivery: if you tie in expected value with deliverables, you empower and enable engineering teams to smartly cut scope and deliver in shorter amount of time the biggest chunk of the value. It's usually only described as "cutting scope", but it requires real creativity and smarts on the engineering team (they can best tell what's feasible in remaining time, when they have tackled all the unknown unknowns you bring up) to do that in a way that still brings the most of the original value. If you move this to your "renegotiation" step, it becomes too slow and hurts delivery as well.

msteffen 2y ago
I basically agree (I think) in that I don’t think there needs to be a separate “cutting scope” phase—one should it as soon as they realize they need to. And I certainly agree that it’s incumbent on engineering to invent alternatives and propose them when we realize that we need to cut scope (for the reasons you mentioned). FWIW, though, engineers shouldn’t decide what to cut (i.e. choose alternatives) in a vacuum, in my experience. I’ve been in plenty of meetings with product/sales/support where they say “it’s better for the whole project to slip than to release it without this one detail” or “we would give up what seems like basic usability to get one particular piece of polish”
- necovek 2y ago
  Yes, totally agreed that engineers should not "decide" in a vacuum. But if they understand what value was supposed to be brought, have a decent product/customer focus, and are creative enough, they can propose good alternatives and not stall the delivery. I've also experienced things you mention, but it was always in orgs where everything, including any small feature, was treated as a "big bet": unsupported by metrics (no matter how imperfect), but instead wishful thinking that it will bring meaningful improvement. As such, you can't come up with anything that's an equally good improvement with less effort because there is no baseline to compare against.
  - msteffen 2y ago
    > I've also experienced things you mention, but it was always in orgs where everything, including any small feature, was treated as a "big bet": unsupported by metrics (no matter how imperfect), but instead wishful thinking that it will bring meaningful improvement. As such, you can't come up with anything that's an equally good improvement with less effort because there is no baseline to compare against. Interesting, that’s exactly the situation I was in, but I never connected the lack of metrics to these kinds of requests. TIL. I feel like I have lot to say about how this manifested. Product direction was very heavily guided by existing customers (because support could say “we have these three customers asking for X”), somewhat guided by closing deals (because sales could say “we have a $$$ deal that the customer says will close if we deliver Y”) and hardly guided at all by the broader market, because product’s suggestions could only ever be supported by speculation and vibes. But we were B2B, so I don’t even know what good metrics would’ve looked like—it’s not like we had billions of users

rixed 2y ago

I find your framing interesting. But most importantly, even if nothing changes in the requirements or the team, the mere new information that is learned while implementing the project is often enough to change the estimate considerably.

msteffen 2y ago
Yes! In my last role, I aggressively pushed the concept of “technical risk” to try and get people used to this idea. Sometimes the computers don’t work the way you thought they did at the beginning. This is a normal and ubiquitous form of risk that just needs to be managed like any other type of risk, with prototypes, multiple revisions of the implementation (each introducing additional risk), occasionally re-scoping, etc. People outside engineering sometimes don’t like it because the risk is human in origin—it comes from engineers not knowing things, and our job is to know things—but until there’s an omniscient engineer, this risk will continue to exist.
- rixed 2y ago
  Yes, this fits my experience as well. Oftentimes I've heard people who are not and have never been software engineers ask "When we want to built a bridge, engineers can tell how long it will take and how expensive it will be, how come with software engineers it's never the case?" (often implied: you software engineers have it easy, slackers! -- I believe we put this blame on ourselves with our immature culture, but that's another discussion) What I usually answer to that is that, to begin with, that if they had actually dealed with big construction projects like bridges then they would not idealize those so much. And secondly, that this is not the right analogy. Bridge construction is a much stabler science, it had no "complexity explosion". A better analogy would be geophysical prospection : the theory it's based on is sound and mature but the unknowns dominate everything in predicting the outcome.
  - msteffen 2y ago
    If you haven’t seen it (and need something to show the “what about bridges” crew), Hillel Wayne’s “crossover project” series of posts[^1] on this is brilliant. I like to show people this quote from it: > One person talked about how frustrating it is to start work on a bridge foundation, only to find that this particular soil freezes in a weird way that makes it liquefy too much in an earthquake. Back to the drawing board. [^1]: https://www.hillelwayne.com/post/we-are-not-special/ https://www.hillelwayne.com/post/we-are-not-special/

nine_zeros 2y ago

- the way to manage this risk is to include buffer in the project timeline, and, critically, to use that buffer not for development tasks that were harder than expected, but for re-planning and re-negotiating the deliverables with stakeholders, to un-explode the project. Agile builds this re-evaluation into the project lifecycle, which is the main (maybe only?) thing is does really right. This is the only right answer. Unfortunately, engineering leadership is too detached from understanding how software works. They think of projects like contractor painting jobs. So easy to estimate - sq. ft * number of people * number of hours. But software is not like that. Software is constantly changing every hour, every day. As a result, any estimation is changing every hour every day. The best thing is for leadership to start acknowledging the realm they are dealing with. If they can't they should step down. But realistically, engineers should always give 400% padding with estimates. The root cause of this estimation is poor leadership. It is not engineering team's problem that detached management doesn't understand software.

wkirby 2y ago

It's been several years since my colleague put this post together, but I think for us it's proven to be a very constructive framework for talking to clients. It's probably worth calling out that we are, specifically, outside contractors providing estimates for clients --- usually clients at the start of their software development journey. I think a lot of our framework holds for developers working on in-house engineering teams, but there are bound to be some differences. That said, I'll be curious to read your post whenever it goes up. Namely because I don't know what the conclusion of your comment is. Whether it's as a vendor or as an engineer on staff, estimates are a hard requirement of software development. Most stakeholders are not developers, and you can only educate the recipients of your estimates so much on the nuances of what is and isn't hard to accommodate. I don't really disagree with any of your assertions --- but isn't the re-evaluation process simply... more estimation? Aren't accounting for project risks --- like which developer is going to pick up the task, what requirements are likely or unlikely to change --- part of producing a quality estimate? Communicating expectations is hard, and to me, one of the defining lines between junior and senior developers is the ability to clearly account for expected risks, and identify plausible but unlikely risks, and to incorporate mitigation into the plan of attack.

msteffen 2y ago
I agree, and in general I am very pro-estimates (they switch your team from cooperative scheduling to preemptive scheduling, and they’re an important tool for managing the chaos of development), but the statement “we expect this project to take eight weeks, but it could take 24 weeks in the worst case” is one I would be very loathe to make. Granted, I’m coming from a B2B startup rather than a consultancy, so I was working with a mixture of internal (product, sales, support) and external (customers) stakeholders. The perspective I would try to give people, though, was this: If we discover a bomb partway through the project (oops, groups are limited to 100 members in this auth system, so we will need to find a new system and migrate or else not support groups) then 8 weeks will no longer be enough, but maybe neither is 24. The right thing for us to do in that situation, IMO, is go back to the stakeholders and align on a new plan. Maybe groups will come in v2, or some groups will work and some won’t, or maybe we do the migration and add it to the estimate, or whatever. One shouldn’t use the 16 buffer weeks between 8 and 24 to sneak in a backend migration (which itself might be a 16-week project, or might not), but to pause and re-align on a plan everyone likes. When I was giving estimates, I would try to frame them as “we’ll spend eight weeks trying to accomplish this list of things. If we discover an issue that prevents us from finishing the list for any reason, we’ll come back to you as early as possible with some alternative proposals and re-assess”. Sometimes people felt that this meant we just weren’t willing to commit to our estimates, and this idea of “software development is chaotic” is what I would say, to explain that the issue wasn’t a lack of motivation, but an inescapable knowledge risk that needed to be managed.