6 ms·

An estimate is usually the sum of a bunch of smaller estimates. Those estimates have a probability distribution that is asymmetric. While sometimes they can be

by harry8 5y ago

An estimate is usually the sum of a bunch of smaller estimates. Those estimates have a probability distribution that is asymmetric. While sometimes they can be quicker than anticipated, the lowest they can go is zero (turns out we don't need it, cut it) and that kind of thing is relatively less common than a blowout the other way. Those blowouts are uncapped, while zero time is a minimum there's no maximum. The unknown unknowns are the things that get you. Seems like it will take X but the frobnicator was broken in ways we couldn't have anticipated so we had to get a different thing and then hack it to make it do what we actually needed etc etc. Each individual item has a low probability of that happening to some extent. When you have a bunch of things in sequence one of them probably will if not more. IIRC each item is described by a poisson distribution. (Have I got that right, mean, long tail). Nobody models estimates as a sequence of dependent, poisson distributed events, that I've heard of at least. If you did you might at least get a range on the estimate. Between 2 and 20 days. That would be more accurate. Useful to anyone much? Unclear.

virgilp 5y ago
I tried to do that! Didn't use Poisson, used normal distribution, but I think that's not so relevant - at the end of the day it's still just more elaborate guesswork. I think what's useful is thinking about the problem, and communicating in group/ disseminating the information (this is required in order to do the typical "planning poker" or whatever planning technique the team happens to use). Whether you end up with L/ S /M or 7/13/21 points or whatever else you use - it's irrelevant, so no need to sweat it; what is relevant is that you discussed the tasks and everyone has some understanding, and maybe they're a little bit better spec-ed now.
- samus 5y ago
  Normal distribution is deeply inapproprite for time estimation because it yields the same probability for 3x the estimated time than for the negative amount(!) of the estimated time. Your second point rings closer to what we should be doing instead. We should put more effort into analysis and refinement. In my experience, estimation meetings where all tasks got estimated were almost useless in the end; and the most valuable meetings where when we didn't finish to estimate, but instead gained a deeper understanding of the requirements and of the domain.
morelandjs 5y ago
Exactly. One of the fundamental tenets of science is reporting estimates with appropriate precision, e.g. significant digits. If the probabilistic interquartile range spans multiple months, why would I report the mean of the estimate in "hours of effort"? What's frustrating is I've articulated these thoughts clearly to managers in the past. They nod their head and agree and then ask for the hourly estimate again and we all collectively share our disappointment when our estimate is inaccurate after the fact. Nothing like running head first into a wall and expecting a different result. My management suggestion? Frequently identify and advocate for simplifications, and modulate the degree of simplification based on progress to date. If people enjoy their work, they will work hard regardless of timeline hawking. So the variable factor is really the volume and complexity of work, not worker intensity or motivation.
- deleted 5y ago
  [deleted]
- gatlin 5y ago
  Yes that would mean managers have to do something, and while some are incidentally useful and productive, there are few external incentives for them to be so. Yet there are many incentives to let you burn out after hitting a wall a hundred times.
eutectic 5y ago
The poisson distribution is quite-light tailed. With a poisson distribution, the probability of completing the task per unit time is constant. Real world tasks are likely to take longer if they have already gone on for longer; maybe a Pareto distribution would be more appropriate.
- da39a3ee 5y ago
  I think you meant exponential distribution? Poisson is a discrete distribution ( number of task completion events occurring in a given time interval if task completion is exponential)
danpalmer 5y ago
For projects with a deadline that matters, I've found that explicitly estimating to "80%" likelihood has helped me hit deadlines much more consistently. I used this to hit a deadline on a 4 month software project only 10 days out, which I think is good for software. I think implicitly most estimates are the 50/50 case. "I'm 50% sure I can do it in this time". Much of the time I don't think this is very useful, and isn't very well thought through.
- noduerme 5y ago
  You have to assume you're going to hit a brick wall at least a few times in any given project, and something you thought would be a snap turns out to require days of refactoring. Especially if there's some aspect of business logic that wasn't clearly communicated beforehand and/or the customer hadn't considered until it came to light as a code problem. This should be built in to the estimate, but always come as a surprise gift to the customer when you still manage to deliver it on time. In design and code, customers need to be consulted and asked what they wanted and made to feel important but not about anything actually mission critical. That's what you're hired for. At least a few times in any long project you need to come up with a meaningful sounding but essentially empty aesthetic decision for the customer to make. This allows you to both show progress and let them feel they're in control. It also allows you to extend the deadline if you feel yourself behind.
- regularfry 5y ago
  50% is optimistic. Some (ok, many) estimates are actually 1% projections. The soonest we could possibly be done. This, too, is not very useful.
HPsquared 5y ago
Lognormal distribution is always a good one. It's applicable where the randomness has a multiplicative rather than additive effect. For example, the work may double or be halved. Lots of natural processes work this way. https://en.m.wikipedia.org/wiki/Log-normal_distribution https://en.m.wikipedia.org/wiki/Log-normal_distribution
- marcosdumay 5y ago
  An useful property is that the Law of Large Numbers doesn't apply to it, so all of your estimate history is basically useless. Also, when you add enough estimates, the total time to completion is mostly determined by the error in one or two of them.
- jbay808 5y ago
  I've written about this before more extensively, but my previous company tracked time vs predictions very rigorously and extensively, and they always fit a lognormal distribution shockingly well.
da39a3ee 5y ago
So a Poisson distribution is for count data, not continuous like time. A gamma distribution is a continuous distribution that have the shape you want. But, this approach should involve getting data and plotting it to see empirically what sort of distribution it might have. I’m also unclear how the dependency you mention is going to come in. That said I agree with you in general that some statistical reasoning should be brought to this problem. Maybe just showing people the way variances are additive and the impact that has on the overall variance of a project made of many small projects. You could pick a plausible gamma distribution for individual task duration and play around with seeing what the sum of a number of IID gammas looks like —- the sum is a gamma distribution also; see wikipedia but the formula.
wpietri 5y ago
And one of the classic problems with estimates is that common processes tend to bias estimates. For example, what (I think) McConnell calls "picking the first non-impossible date". An exec asks how long project Magic Pony will take. A manager asks the team and the team comes up with a number. If the number is higher than what the exec imagined, or even higher than what the manager imagines the exec imagines, pressure is applied on the number. "Are you sure? That seems like a lot." A negotiation ensues, and often the number lands on the first date that engineers can't absolutely prove is impossible. Or the first date that engineers won't quit their jobs over. For an estimate, we in theory want one where the team has a good chance of hitting it. Say, a 75% chance of success. But iterative pressure from the powerful will shift the distribution to 25%, 5%, 1%.
- quacked 5y ago
  The classic cognitive trick--people, unless specifically trained not to, will always estimate the best possible scenario, even if past experience has told them that the best possible scenario never happens. I'm dealing with this at my company right now. Our clients, and even our internal teams, will always refuse to estimate according to our past behavior. Us: "We estimate your build will be functional in prod in five months." Client: "Unacceptable. That's way too long. We need it in three." Us: "Okay, we technically can do that, but that relies on a big internal release that we think is risky to base estimates on." Client: "All I'm hearing is that you can do it in three." Then we get to three months and hit delays over and over again until finally, five months later, the build is functional in prod. The thing that really frustrates me is that when we deliver the initial estimate, the clients always refuse to accept our medium-case estimate, with the implication that they'll drop us as a vendor if we follow that. So we give them our best-case estimate, fail to hit it, and then they buy more from us anyway, complaining the entire time. Why don't we all just look at the past 10 builds we delivered for them, admit that the best-case never happens, and stop pretending like it's unacceptable to take longer? The point here is that they keep accepting it. Everyone complains, but we all keep selling and buying to and from each other. Let's just admit it's harder than we're saying to get software built and accept longer estimates, given that we're all going to keep selling and buying anyway.
snarf21 5y ago
Product and sales always want exact dates from tech but always seem to be a lot more vague when it comes to customer numbers and revenue numbers. I've found the following system to work well. Estimates are always given as the following: hours, days, weeks, months, quarters. That's it, not numbers attached. It means "some small number of _______". This allows for discussion without everyone feeling trapped. If a feature takes "weeks" and Product thinks it should take "hours", then that is a discussion that can allow for a change in scope or requirement clarification or just education about how the system works. Of course, PjM systems only work if everyone buys in and that is always the bigger challenge.
AnimalMuppet 5y ago
> Useful to anyone much? Unclear. If you're the leader/manager/person in charge, do you want to see reality, or do you want to keep your head in the sand so you can have a simpler planning process? If you want reality, and that's what the most accurate view of reality, then by definition it's more useful than anything else.
ngrilly 5y ago
Is there a way do calculate this in Excel without using a Monte Carlo simulation?

by harry8 5y ago

virgilp 5y ago

I tried to do that! Didn't use Poisson, used normal distribution, but I think that's not so relevant - at the end of the day it's still just more elaborate guesswork. I think what's useful is thinking about the problem, and communicating in group/ disseminating the information (this is required in order to do the typical "planning poker" or whatever planning technique the team happens to use). Whether you end up with L/ S /M or 7/13/21 points or whatever else you use - it's irrelevant, so no need to sweat it; what is relevant is that you discussed the tasks and everyone has some understanding, and maybe they're a little bit better spec-ed now.

samus 5y ago
Normal distribution is deeply inapproprite for time estimation because it yields the same probability for 3x the estimated time than for the negative amount(!) of the estimated time. Your second point rings closer to what we should be doing instead. We should put more effort into analysis and refinement. In my experience, estimation meetings where all tasks got estimated were almost useless in the end; and the most valuable meetings where when we didn't finish to estimate, but instead gained a deeper understanding of the requirements and of the domain.

morelandjs 5y ago

Exactly. One of the fundamental tenets of science is reporting estimates with appropriate precision, e.g. significant digits. If the probabilistic interquartile range spans multiple months, why would I report the mean of the estimate in "hours of effort"? What's frustrating is I've articulated these thoughts clearly to managers in the past. They nod their head and agree and then ask for the hourly estimate again and we all collectively share our disappointment when our estimate is inaccurate after the fact. Nothing like running head first into a wall and expecting a different result. My management suggestion? Frequently identify and advocate for simplifications, and modulate the degree of simplification based on progress to date. If people enjoy their work, they will work hard regardless of timeline hawking. So the variable factor is really the volume and complexity of work, not worker intensity or motivation.

deleted 5y ago
[deleted]
gatlin 5y ago
Yes that would mean managers have to do something, and while some are incidentally useful and productive, there are few external incentives for them to be so. Yet there are many incentives to let you burn out after hitting a wall a hundred times.

eutectic 5y ago

The poisson distribution is quite-light tailed. With a poisson distribution, the probability of completing the task per unit time is constant. Real world tasks are likely to take longer if they have already gone on for longer; maybe a Pareto distribution would be more appropriate.

da39a3ee 5y ago
I think you meant exponential distribution? Poisson is a discrete distribution ( number of task completion events occurring in a given time interval if task completion is exponential)

danpalmer 5y ago

For projects with a deadline that matters, I've found that explicitly estimating to "80%" likelihood has helped me hit deadlines much more consistently. I used this to hit a deadline on a 4 month software project only 10 days out, which I think is good for software. I think implicitly most estimates are the 50/50 case. "I'm 50% sure I can do it in this time". Much of the time I don't think this is very useful, and isn't very well thought through.

noduerme 5y ago
You have to assume you're going to hit a brick wall at least a few times in any given project, and something you thought would be a snap turns out to require days of refactoring. Especially if there's some aspect of business logic that wasn't clearly communicated beforehand and/or the customer hadn't considered until it came to light as a code problem. This should be built in to the estimate, but always come as a surprise gift to the customer when you still manage to deliver it on time. In design and code, customers need to be consulted and asked what they wanted and made to feel important but not about anything actually mission critical. That's what you're hired for. At least a few times in any long project you need to come up with a meaningful sounding but essentially empty aesthetic decision for the customer to make. This allows you to both show progress and let them feel they're in control. It also allows you to extend the deadline if you feel yourself behind.
regularfry 5y ago
50% is optimistic. Some (ok, many) estimates are actually 1% projections. The soonest we could possibly be done. This, too, is not very useful.

HPsquared 5y ago

Lognormal distribution is always a good one. It's applicable where the randomness has a multiplicative rather than additive effect. For example, the work may double or be halved. Lots of natural processes work this way. https://en.m.wikipedia.org/wiki/Log-normal_distribution https://en.m.wikipedia.org/wiki/Log-normal_distribution

marcosdumay 5y ago
An useful property is that the Law of Large Numbers doesn't apply to it, so all of your estimate history is basically useless. Also, when you add enough estimates, the total time to completion is mostly determined by the error in one or two of them.
jbay808 5y ago
I've written about this before more extensively, but my previous company tracked time vs predictions very rigorously and extensively, and they always fit a lognormal distribution shockingly well.

da39a3ee 5y ago

So a Poisson distribution is for count data, not continuous like time. A gamma distribution is a continuous distribution that have the shape you want. But, this approach should involve getting data and plotting it to see empirically what sort of distribution it might have. I’m also unclear how the dependency you mention is going to come in. That said I agree with you in general that some statistical reasoning should be brought to this problem. Maybe just showing people the way variances are additive and the impact that has on the overall variance of a project made of many small projects. You could pick a plausible gamma distribution for individual task duration and play around with seeing what the sum of a number of IID gammas looks like —- the sum is a gamma distribution also; see wikipedia but the formula.

wpietri 5y ago

And one of the classic problems with estimates is that common processes tend to bias estimates. For example, what (I think) McConnell calls "picking the first non-impossible date". An exec asks how long project Magic Pony will take. A manager asks the team and the team comes up with a number. If the number is higher than what the exec imagined, or even higher than what the manager imagines the exec imagines, pressure is applied on the number. "Are you sure? That seems like a lot." A negotiation ensues, and often the number lands on the first date that engineers can't absolutely prove is impossible. Or the first date that engineers won't quit their jobs over. For an estimate, we in theory want one where the team has a good chance of hitting it. Say, a 75% chance of success. But iterative pressure from the powerful will shift the distribution to 25%, 5%, 1%.

quacked 5y ago
The classic cognitive trick--people, unless specifically trained not to, will always estimate the best possible scenario, even if past experience has told them that the best possible scenario never happens. I'm dealing with this at my company right now. Our clients, and even our internal teams, will always refuse to estimate according to our past behavior. Us: "We estimate your build will be functional in prod in five months." Client: "Unacceptable. That's way too long. We need it in three." Us: "Okay, we technically can do that, but that relies on a big internal release that we think is risky to base estimates on." Client: "All I'm hearing is that you can do it in three." Then we get to three months and hit delays over and over again until finally, five months later, the build is functional in prod. The thing that really frustrates me is that when we deliver the initial estimate, the clients always refuse to accept our medium-case estimate, with the implication that they'll drop us as a vendor if we follow that. So we give them our best-case estimate, fail to hit it, and then they buy more from us anyway, complaining the entire time. Why don't we all just look at the past 10 builds we delivered for them, admit that the best-case never happens, and stop pretending like it's unacceptable to take longer? The point here is that they keep accepting it. Everyone complains, but we all keep selling and buying to and from each other. Let's just admit it's harder than we're saying to get software built and accept longer estimates, given that we're all going to keep selling and buying anyway.

snarf21 5y ago

Product and sales always want exact dates from tech but always seem to be a lot more vague when it comes to customer numbers and revenue numbers. I've found the following system to work well. Estimates are always given as the following: hours, days, weeks, months, quarters. That's it, not numbers attached. It means "some small number of _______". This allows for discussion without everyone feeling trapped. If a feature takes "weeks" and Product thinks it should take "hours", then that is a discussion that can allow for a change in scope or requirement clarification or just education about how the system works. Of course, PjM systems only work if everyone buys in and that is always the bigger challenge.

AnimalMuppet 5y ago

> Useful to anyone much? Unclear. If you're the leader/manager/person in charge, do you want to see reality, or do you want to keep your head in the sand so you can have a simpler planning process? If you want reality, and that's what the most accurate view of reality, then by definition it's more useful than anything else.

ngrilly 5y ago

Is there a way do calculate this in Excel without using a Monte Carlo simulation?