4 ms·

> ...if you want to make a commitment you can take to a customer, you can just extend the lognormal up to... What you're proposing is effectively a fancy "padd

> ...if you want to make a commitment you can take to a customer, you can just extend the lognormal up to... What you're proposing is effectively a fancy "padding factor", it's like on Star Trek when Scotty says they can only go at Warp-8, but really, Warp-11 is possible. Sometimes stakeholders do behave like toddlers and get red in the face and stomp around when a "promise" is broken because "we aren't there yet." (ok, maybe not literally, but whatever the adult version of a toddler flip-out is). But most of the time people can understand that unexpected problems come up. It's perfectly fine to make log-normal plots and think about statistics, but everyone will much happier if you explain what's going on to the stake-holders instead of just giving "black-box" estimates using confidence intervals.

kqr 6y ago
I disagree somewhat. There's a world of difference between saying "here's when we think it'll be done, but sometimes shit happens and there'll be a delay" and saying "we're 95 % confident it will be done by this date" -- especially when you have a solid track record behind the second. The first one tells you effectively nothing. The second one you can build strategy on.
- crispyambulance 6y ago
  > "here's when we think it'll be done, but sometimes shit happens and there'll be a delay" I agree that a statement like that would not be useful. What I am saying is that it's better to make a good effort at estimation but then keep the stake-holders informed about what is going on, about the challenges and the mid-project delays. Any "95%" confidence calculation is kinda dubious. It would only work if you have meticulous and accurate records, have a long-enough history of delivering the same kind of product with the same team, the same customer-base, and the same tooling, oh yeah, and everyone has the psychological safety factor to be honest. If you work in a place where that exists, more power to you, but I think that's rare.
  - kqr 6y ago
    I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention. It does take good management/culture, but if I don't even have that, I'm headed for the dumpster anyway and should go home and rethink the whole thing first. > What I am saying is that it's better to make a good effort at estimation But what does that estimation mean? If we don't know after the fact whether the estimation was good or bad, then it's useless because it's an answer to an unknown question, and every stakeholder will interpret it differently. In order to know after the fact, we must know ahead of time whether we're aiming for 50 %, 90 %, or something else.
    - crispyambulance 6y ago
      > I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention. I doubt it. It's dicey to do estimations even when everything is well understood, but if you lack detailed and accurate records of past work for very similar projects... the best you can do is pad like crazy, cross your fingers, meet the deadline and say your confidence interval calculation was "accurate."
      - jbay808 6y ago
        If the lognormal distribution is correct, the 95% confidence interval is a bit over 5 times the developer's gut-feeling estimate. The 99% confidence interval is 10 times the developer's gut-feel estimate. We found a lot of data to back this up, but your mileage may vary.
kybernetikos 6y ago
This is a principled approach. And of course it does depend on the nature of the contractual relationship with your client. However, even if we charitably assume that the developers are accurately estimating the mean, that means that out of 100 projects, 50 of them will be delivered late. It is likely that you'd want to do better than that, and to do so in a principled manner, you're going to need to adjust the estimates that come out of the estimation process.
jbay808 6y ago
The idea here is just to be precise in what we're communicating, when providing a forecast, because different departments have different risk tolerances on that forecast. What the lognormal says is "it might even be done by late May, most likely by June, but almost certainly by December". The dev department needs to allocate people, so for them, the most useful thing is the mean completion time. So for them, June is the important thing, although they want to be aware of potential delays in case they need to reshuffle. The sales department is playing a very different game. Upset, impatient customers cost a lot, both in terms of sales and reputation, and can be very unforgiving. Sales much prefers to communicate a hugely pessimistic version of when something will be ready, that they have confidence in, and possibly be early, rather than a 50-50 estimate. This is decision theory in action, where each actor combines the same probability distribution with different payoff matrices and so arrives at different conclusions suitable to their role. To enable this, we clarified the language around "estimates", "commitments", and "promises", as shorthand for different intervals of the same underlying distribution.

kqr 6y ago

I disagree somewhat. There's a world of difference between saying "here's when we think it'll be done, but sometimes shit happens and there'll be a delay" and saying "we're 95 % confident it will be done by this date" -- especially when you have a solid track record behind the second. The first one tells you effectively nothing. The second one you can build strategy on.

crispyambulance 6y ago
> "here's when we think it'll be done, but sometimes shit happens and there'll be a delay" I agree that a statement like that would not be useful. What I am saying is that it's better to make a good effort at estimation but then keep the stake-holders informed about what is going on, about the challenges and the mid-project delays. Any "95%" confidence calculation is kinda dubious. It would only work if you have meticulous and accurate records, have a long-enough history of delivering the same kind of product with the same team, the same customer-base, and the same tooling, oh yeah, and everyone has the psychological safety factor to be honest. If you work in a place where that exists, more power to you, but I think that's rare.
- kqr 6y ago
  I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention. It does take good management/culture, but if I don't even have that, I'm headed for the dumpster anyway and should go home and rethink the whole thing first. > What I am saying is that it's better to make a good effort at estimation But what does that estimation mean? If we don't know after the fact whether the estimation was good or bad, then it's useless because it's an answer to an unknown question, and every stakeholder will interpret it differently. In order to know after the fact, we must know ahead of time whether we're aiming for 50 %, 90 %, or something else.
  - crispyambulance 6y ago
    > I think I can work out a 95 % estimation I am willing to stake money on even without the things you mention. I doubt it. It's dicey to do estimations even when everything is well understood, but if you lack detailed and accurate records of past work for very similar projects... the best you can do is pad like crazy, cross your fingers, meet the deadline and say your confidence interval calculation was "accurate."
    - jbay808 6y ago
      If the lognormal distribution is correct, the 95% confidence interval is a bit over 5 times the developer's gut-feeling estimate. The 99% confidence interval is 10 times the developer's gut-feel estimate. We found a lot of data to back this up, but your mileage may vary.

kybernetikos 6y ago

This is a principled approach. And of course it does depend on the nature of the contractual relationship with your client. However, even if we charitably assume that the developers are accurately estimating the mean, that means that out of 100 projects, 50 of them will be delivered late. It is likely that you'd want to do better than that, and to do so in a principled manner, you're going to need to adjust the estimates that come out of the estimation process.

jbay808 6y ago

The idea here is just to be precise in what we're communicating, when providing a forecast, because different departments have different risk tolerances on that forecast. What the lognormal says is "it might even be done by late May, most likely by June, but almost certainly by December". The dev department needs to allocate people, so for them, the most useful thing is the mean completion time. So for them, June is the important thing, although they want to be aware of potential delays in case they need to reshuffle. The sales department is playing a very different game. Upset, impatient customers cost a lot, both in terms of sales and reputation, and can be very unforgiving. Sales much prefers to communicate a hugely pessimistic version of when something will be ready, that they have confidence in, and possibly be early, rather than a 50-50 estimate. This is decision theory in action, where each actor combines the same probability distribution with different payoff matrices and so arrives at different conclusions suitable to their role. To enable this, we clarified the language around "estimates", "commitments", and "promises", as shorthand for different intervals of the same underlying distribution.