5 ms·

This is basically your solution: http://imgur.com/YNmKId9 http://imgur.com/YNmKId9 PHB: Use the CRS database to size that market. Dilbert: That data is wrong.

This is basically your solution: http://imgur.com/YNmKId9 http://imgur.com/YNmKId9 PHB: Use the CRS database to size that market. Dilbert: That data is wrong. PHB: Then use the SIBS database. Dilbert: That data is also wrong. PHB: Can you average them? Dilbert: Sure. I can multiply them too. ----- I kid (sorta). It misses the core issue, why are developers estimates so terrible, 3 terrible estimates don't make a "good" one. The post hits on some of the points, but IMHO misses a major one. Estimations are based on experience, yet developers are constantly building brand new types of products. If all you built was "blog engines" over and over again, you would get damned good at estimating them. The trick in software is as soon as something needs to be reused over and over again, it is abstracted, put in a library, built into a framework, so you can make your product different (read: build something entirely new) -- and EVERYONE sucks at estimating the cost of building things entirely new. Think of the (entirely new in 1962) Lunar Module... they thought it would cost at max 350 million. It cost 2.2 billion. Those where very smart people with a major stake in getting the initial number right, but it was just a crazy ass guess because they had NEVER BUILT ONE BEFORE.

lobotryas 13y ago
You are absolutely right. It's impossible to estimate how long it will take to complete something you have never done before in your life. Unless the new code being written is absolutely trivial, estimates are going to be guesses, at best. The only half-decent solution I found so far is iterative development. Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete. Only after completing your first "big rock" in this way can you use your new data to gauge your velocity and give an estimate for completing the rest of the work that's worth the paper it's written on.
- jacques_chester 13y ago
  > Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete. Experimental psychologists have found that decomposing tasks increases estimate accuracy, regardless of how or why the task is decomposed. It's called the "unpacking effect". http://www.sciencedirect.com/science/article/pii/S002210310300177X http://www.sciencedirect.com/science/article/pii/S0022103103...
  - webhat 13y ago
    Unpacking works great when it's clear what the tasks are going to be. In my experience the bulk of the work does not go into the project or tool itself, i.e. completing the tasks, but rather into resolving issues. I usually estimate that writing the code is anywhere from 10-20% of completing the project. Some research shows that 1 kloc of code contains between 15 and 50 defects, and that code reviewing will pick up only 70-90%. This still leaves a lot of bugs. I sometime ago wrote up some meta research on estimating code reviewing: http://specialbrands.net/2011/10/24/code-review-in-scrum-xp-agile-mathematics/ http://specialbrands.net/2011/10/24/code-review-in-scrum-xp-...
    - jacques_chester 13y ago
      > I usually estimate that writing the code is anywhere from 10-20% of completing the project. McConnell's book has a really excellent table of items that are left out of estimates. Nuts-and-bolts things like "API documentation", "deployment scripts", "status emails" and so on. > This still leaves a lot of bugs. Right. The classical software engineering theory is to build multiple quality gates and to study them for defects yielded. I recall that inspection stomps everything else; automatic testing came second. The problem of time-to-fix is subtly different from time-to-release, though. Software with known defects is regularly in use.
- kabdib 13y ago
  Decomposing as far as you can is a good practice ... you'll still be way off, because you think you know all the problems that you're going to need to solve. So about halfway through the project your list is going to be longer and management is going to be freaking out. The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?" Sometimes they are very easy. Sometimes they are very hard. For the really hard bugs, you need to be clever, scientific, determined, methodical and LUCKY. I don't know how to know when I'm going to run into some luck.
  - jacques_chester 13y ago
    > you'll still be way off, because you think you know all the problems that you're going to need to solve Absolutely; but I think it's reasonable to assume that a more accurate estimate is a more valuable estimate. You can also track estimate error over time and use that to adjust future estimates. > The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?" The problem is that in any sample of n=1, variability is huge. It's like saying "I am going to pick one person out of the global population. How tall is he or she?" Well of course I can't know that. The sample is too small. But if instead I am told "we are going to pick ten thousand people at random from the global population; how tall are they in total?" then I can use previously-gathered statistics to give a range of estimates that will probably be accurate.
ExpiredLink 13y ago
> Estimations are based on experience, yet developers are constantly building brand new types of products. If all you built was "blog engines" over and over again, you would get damned good at estimating them. The trick in software is as soon as something needs to be reused over and over again, it is abstracted, put in a library, built into a framework, so you can make your product different (read: build something entirely new) -- and EVERYONE sucks at estimating the cost of building things entirely new. That's it. Most, actually all, literature on software estimation I've seem ignores this simple fact. It's feasible to estimate cost and time for building houses, even skyscrapers, if you do that all the time. But how long does it take to climb a never-climbed mountain?
- phillmv 13y ago
  And even then, large capital projects are routinely wildly off estimates. I think the delta in "how far off we were" just tends to be larger more frequently.
  - jacques_chester 13y ago
    In the Industrial Megaprojects book I quoted elsewhere, the single best predictor of every measure of project performance was "Front End Loading" -- how completely the project has been studied and specified before it is funded. There are at least three FEL stages. Companies that muck up FEL-1 are doomed to failure, companies that muck up FEL-2 or FEL-3 are doomed to poor returns.
- jacques_chester 13y ago
  > But how long does it take to climb a never-climbed mountain? This estimate could be derived in a number of ways. By analogy to other mountains, by creating a statistical model of climbing time based on parameters like gradient and climber skill, or by aggregating expert judgements of the mountain's likely climbing time. A better example of a difficult-to-estimate task was the Manhattan Project. How long would it take to build the first atomic weapon? That was a truly novel problem, insofar as it would require fundamental physics research to solve. But almost none of our work is of that nature. To be sure, there are unexpected requirements and complexities, but we are almost never performing fundamental research to achieve our ends.

lobotryas 13y ago

You are absolutely right. It's impossible to estimate how long it will take to complete something you have never done before in your life. Unless the new code being written is absolutely trivial, estimates are going to be guesses, at best. The only half-decent solution I found so far is iterative development. Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete. Only after completing your first "big rock" in this way can you use your new data to gauge your velocity and give an estimate for completing the rest of the work that's worth the paper it's written on.

jacques_chester 13y ago
> Take a feature and break it down i to big rocks. Further break down the big rocks into little chunks - each that you estimate would take you less than a day to complete. Experimental psychologists have found that decomposing tasks increases estimate accuracy, regardless of how or why the task is decomposed. It's called the "unpacking effect". http://www.sciencedirect.com/science/article/pii/S002210310300177X http://www.sciencedirect.com/science/article/pii/S0022103103...
- webhat 13y ago
  Unpacking works great when it's clear what the tasks are going to be. In my experience the bulk of the work does not go into the project or tool itself, i.e. completing the tasks, but rather into resolving issues. I usually estimate that writing the code is anywhere from 10-20% of completing the project. Some research shows that 1 kloc of code contains between 15 and 50 defects, and that code reviewing will pick up only 70-90%. This still leaves a lot of bugs. I sometime ago wrote up some meta research on estimating code reviewing: http://specialbrands.net/2011/10/24/code-review-in-scrum-xp-agile-mathematics/ http://specialbrands.net/2011/10/24/code-review-in-scrum-xp-...
  - jacques_chester 13y ago
    > I usually estimate that writing the code is anywhere from 10-20% of completing the project. McConnell's book has a really excellent table of items that are left out of estimates. Nuts-and-bolts things like "API documentation", "deployment scripts", "status emails" and so on. > This still leaves a lot of bugs. Right. The classical software engineering theory is to build multiple quality gates and to study them for defects yielded. I recall that inspection stomps everything else; automatic testing came second. The problem of time-to-fix is subtly different from time-to-release, though. Software with known defects is regularly in use.
kabdib 13y ago
Decomposing as far as you can is a good practice ... you'll still be way off, because you think you know all the problems that you're going to need to solve. So about halfway through the project your list is going to be longer and management is going to be freaking out. The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?" Sometimes they are very easy. Sometimes they are very hard. For the really hard bugs, you need to be clever, scientific, determined, methodical and LUCKY. I don't know how to know when I'm going to run into some luck.
- jacques_chester 13y ago
  > you'll still be way off, because you think you know all the problems that you're going to need to solve Absolutely; but I think it's reasonable to assume that a more accurate estimate is a more valuable estimate. You can also track estimate error over time and use that to adjust future estimates. > The elephant in the room: Debugging. I don't know how many times I've been asked, "When are you going to find that bug?" The problem is that in any sample of n=1, variability is huge. It's like saying "I am going to pick one person out of the global population. How tall is he or she?" Well of course I can't know that. The sample is too small. But if instead I am told "we are going to pick ten thousand people at random from the global population; how tall are they in total?" then I can use previously-gathered statistics to give a range of estimates that will probably be accurate.

ExpiredLink 13y ago

> Estimations are based on experience, yet developers are constantly building brand new types of products. If all you built was "blog engines" over and over again, you would get damned good at estimating them. The trick in software is as soon as something needs to be reused over and over again, it is abstracted, put in a library, built into a framework, so you can make your product different (read: build something entirely new) -- and EVERYONE sucks at estimating the cost of building things entirely new. That's it. Most, actually all, literature on software estimation I've seem ignores this simple fact. It's feasible to estimate cost and time for building houses, even skyscrapers, if you do that all the time. But how long does it take to climb a never-climbed mountain?

phillmv 13y ago
And even then, large capital projects are routinely wildly off estimates. I think the delta in "how far off we were" just tends to be larger more frequently.
- jacques_chester 13y ago
  In the Industrial Megaprojects book I quoted elsewhere, the single best predictor of every measure of project performance was "Front End Loading" -- how completely the project has been studied and specified before it is funded. There are at least three FEL stages. Companies that muck up FEL-1 are doomed to failure, companies that muck up FEL-2 or FEL-3 are doomed to poor returns.
jacques_chester 13y ago
> But how long does it take to climb a never-climbed mountain? This estimate could be derived in a number of ways. By analogy to other mountains, by creating a statistical model of climbing time based on parameters like gradient and climber skill, or by aggregating expert judgements of the mountain's likely climbing time. A better example of a difficult-to-estimate task was the Manhattan Project. How long would it take to build the first atomic weapon? That was a truly novel problem, insofar as it would require fundamental physics research to solve. But almost none of our work is of that nature. To be sure, there are unexpected requirements and complexities, but we are almost never performing fundamental research to achieve our ends.