6 ms·

All the research indicates that software developers are terrible at estimating time, but are fairly accurate at estimating relative effort. The only reliable me

All the research indicates that software developers are terrible at estimating time, but are fairly accurate at estimating relative effort. The only reliable method is to break down projects into the smallest deliverable "chunks" that you can, and then estimate the relative effort of those "chunks" compared to actual, concrete tasks that the team has completed in the past. After doing this for a while, you can use these relative effort estimates to project completion dates for new projects. This is what agile "story points" are supposed to represent, but many teams unfortunately end up with things like "2 points = 1 week", which just doesn't work. If you use this method, it's absolutely critical that the team always focus on relative effort, and not on time. If you want it to continue to work, you also have to be very careful how you communicate with team members regarding their progress and estimated completion dates - don't tell them that their component is "late" - this will make them think in terms of time next time you're estimating something, and will ruin your estimates.

nautilus12 7y ago
I think the author is trying to say that we should call a spade a spade and say that by story points they really mean time anyway so might as well just ask for a deadline. I've never worked at a company where points didn't have an equivalent time translation.
- iNate2000 7y ago
  The article didn't mention points at all. It said "agile", but only, like, once. It also didn't mention Evidence-Based Scheduling[1]. Also, the evidence-based-scheduling article doesn't mention points either. [1]https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
  - ldng 7y ago
    Except it fails at step 1. If you have to breakdown a “Ajax photo editor” into 16 hour max chunks, as stated, you need a detailed design. And that takes times in itself. The problem is, often times, the manager want the estimate before you've taken the design time because, to them, it should be included in the estimate.
    - alexhutcheson 7y ago
      Yeah, if you have a non-trivial big project that you don't know how to do yet, then the only thing you should be estimating is "Create a design for project X". After that, you might need new tasks "Create a design for sub-systems Foo and Bar". Until you have spent time on design, there's often no point estimating the total project.
commandlinefan 7y ago
> break down projects into the smallest deliverable "chunks" that you can Step 1: Figure out how to do this Step 2: Do it
- mceachen 7y ago
  Step 3: Discover that what you did was wrong (bad implementation or not what they wanted) Step 4: Go to step 1
- alexhutcheson 7y ago
  "Figure out how to do this" (aka create a high-level design) is a task, and you should be treat it the same way as you treat implementing a "chunk": estimate the effort it takes, prioritize it, and then work on it. The only wrinkle here is that you should not try to estimate any "chunks" of the implementation until you've finished your initial high-level design - there's no point estimating when you don't know what the "chunks" are. For big systems you may need to iterate on this a couple times to design sub-components of the larger system and create prototypes for systems that are critical to the project but might not work, etc. Estimation is incremental - sometimes you need to spend some non-trivial effort gathering the information you need before you can estimate the relative effort in building a system.
  - LoSboccacc 7y ago
    > Figure out how to do this is a task this is the crux of the issue. on average you can estimate six month of work if you spend two weeks chopping it into tiny bits. rarely management allow that to happen and developers are rushed in with aggressive randomness. these "developers are bad at estimating" pieces are mostly people that try to estimate effort from complexity which is the wrong approach a priory because complexity hides the project unknowns, and this methodology, along all other comparative methods, is just a Gaussian curve built out off gut feelings: sometimes converges to a realistic estimate for large enough values, but more often than not the center just shows the developer team bias.
    - xyzzy_plugh 7y ago
jredwards 7y ago
> All the research indicates that software developers are terrible at estimating time, but are fairly accurate at estimating relative effort. I'd be interested in reading the research you're referencing, if you have something specific in mind.
- HeyLaughingBoy 7y ago
  I don't have a specific online reference to cite, but the Software Engineering Institute has a lot of research in this area. Specifically, the PSP (Personal Software Process) suggests using LOC as a proxy for size. e.g., a Small task might be 50 LOC, 200 LOC is medium, 700 is large. Then you can take those T-Shirt sizes and compare them to tasks you have done in the past and use that historical data to predict completion times. I'm simplifying it somewhat (e.g., there are also Complexity inputs), but the process does work if the organization is disciplined enough to follow it.
  - necovek 7y ago
    There is a huge difference between "research" and "here's another idea" — I would personally love to hear about some research backing any estimation methodology. FWIW, I "feel" the big issue is with loosely defined requirements which shift as development progresses.
    - HeyLaughingBoy 7y ago
      PSP/TSP isn't "here's another idea." It's a research-backed software development process. And yes, while unmanaged, changing requirements is a major problem, even with proper requirements management you still have the time estimation problem.
  - lloeki 7y ago
    LOCs don’t work either for more situations than I can count in my experience. I’ve had 50ish LOC stuff that implemented a tricky algorithm that was core to a system and required careful design, implementation and testing, right next to 700ish LOC stuff that was super straightforward and basically just laid itself out.
crdrost 7y ago
It’s a much bigger problem than this, sadly. The fact that you are even saying “If you want it to continue to work, you also have to be very careful how you communicate...” is the problem. The problem is that the way we manage things causes engineers and management to be against each other in principle, so that management is pushing one way and engineering is pushing the other way. Your advice is kind of like marital counseling, “let’s agree to not label Tom the person as an inconsiderate asshole but just talk about how he makes you feel,” “OK, well Tom, I am frustrated by your daily behavior that I perceive as assholish and inconsiderate” and I am sure it helps a little but I mean if you think Tom’s feelings aren’t going to be hurt you’re not giving him much credit. In fact if we were being honest with our managers we would be scandalized. We base our estimates on something like an 80% chance of getting something done (which already improves on the 50% chance by roughly doubling our estimate) and then every engineer I know has some informal rule about doubling or tripling the estimate before they hand it off to their manager. Those managers often add their own 20%. Then their managers see that something is up so they slash 50%. More than half of any project—perhaps even three quarters or more—is estimated as safety-factor. You would expect that if we were getting it wrong, we would be getting it wrong the other way, delivering early. The fact that we are late so often is a scandal in its own right. Adding more time buffer is clearly not helping much, since we have such a surfeit of it already. We must have wasted that safety buffer; there is no other explanation for how it disappeared. And once you have set up these luxurious margins of safety it is already too late. Why is the engineer adding that extra factor 4x or whatever it is? It is because management is already at cross-purposes to engineering. That engineer is adding that safety because they do not feel safe without it. Any insufficiently safe ski-lift in active use will see hundreds of buggy ill-specified implementations of safety harnesses: I will go on this ride but I am taking some precautions. The starting assumption has got to be that it is OK to fail on the deadline. This cannot be because of hidden information but must come as a result of trust: “we are eating into the safety buffer, that is totally OK and that is what it is there for, but I want to do whatever I can now to get you out of meetings and approach-planning and to fill out your timesheets for you and heck, even plan an amazing weekend between you and your fiancee to alleviate relationship stresses—and take any other distractions and loads off of your plate so that I can get your single-pointed focus on this feature and we can finish this.” If management makes themselves that safety harness, then there is no cross-purpose.
crumpets 7y ago
>After doing this for a while, you can use these relative effort estimates to project completion dates for new projects. This doesn't work. The problem is that you get periods where the work is relatively easy overall so 5 points ends up being tasks that take 2 days. Then you pick up new difficult features and estimate everything relative to all of the new hard tasks and you end up with 5 points meaning 5 days. Sure you can gauge relative difficulty of tasks between each other, but in a regular developer's day-to-day life the range should really be 1-100 or so rather than making stuff fit into 12 points or less.
- alexhutcheson 7y ago
  1. Keep your baseline tasks you're comparing against consistent for as long as you can (as long as the team remembers them and agrees on the effort they required). Don't use a task from week 2 to estimate week 3, etc. 2. Don't use a short time interval or moving average to determine your points->time mapping. You're looking for the long-run average mapping, not a specific estimate for this developer today.
  - amatecha 7y ago
    +1 , this guy knows what's up. Relative estimates work really well, from my experience!
- lifeisstillgood 7y ago
  i like the 1-100 idea. so often the same points come back (a world of 1 and 3 pointers) but really have we broken that down? I think we should always have reference tasks in front of us -
- yesbabyyes 7y ago
  I think it's common to use the Fibonacci sequence as the difference between each valid weight increases pretty fast.
simplecomplex 7y ago
> All the research indicates Which research would that be?
singron 7y ago
The relative effort estimation seems to be more-or-less accurate for small chunks. However, at the beginning of a project, you don't know how many small chunks there are. In a small project, you can reasonably plan and estimate most of the chunks ahead of time and (eventually) plot out on a burn down chart that shows roughly when the project will complete. However, in a large project, you can really only plan the beginning of the project this way. Eventually there are too many unknowns and you risk all the downsides of waterfall design. In my experience, a project like this has a burn down chart that goes UP over time (i.e. chunks are added faster than they are completed). This makes estimating the project impossible, especially if the chart is still concave-up (i.e. the rate at which it increases is increasing). If you estimate large chunks, then you are less likely to add them over time, but you also won't be as good at estimating them. I think this is a good method for determining if a project is closer to a 1 year project or a 10 year project though. E.g. if you have 10 large equal chunks in a project and it ends up taking you 6 months to complete the first 1, you probably aren't going to finish by the end of the year, and you should evaluate if you are happy spending 5 years on it before you spend any more time.
- jrochkind1 7y ago
  In the "agile" practices I believe it comes from, I think it was actually basically only intended to answer "How much will we probably get done in the next week" or two or three at most, never to estimate an entire 'project'. Matching your perception of where it works and doesn't. i think? Anyone confirm/deny?

nautilus12 7y ago

I think the author is trying to say that we should call a spade a spade and say that by story points they really mean time anyway so might as well just ask for a deadline. I've never worked at a company where points didn't have an equivalent time translation.

iNate2000 7y ago
The article didn't mention points at all. It said "agile", but only, like, once. It also didn't mention Evidence-Based Scheduling[1]. Also, the evidence-based-scheduling article doesn't mention points either. [1]https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
- ldng 7y ago
  Except it fails at step 1. If you have to breakdown a “Ajax photo editor” into 16 hour max chunks, as stated, you need a detailed design. And that takes times in itself. The problem is, often times, the manager want the estimate before you've taken the design time because, to them, it should be included in the estimate.
  - alexhutcheson 7y ago
    Yeah, if you have a non-trivial big project that you don't know how to do yet, then the only thing you should be estimating is "Create a design for project X". After that, you might need new tasks "Create a design for sub-systems Foo and Bar". Until you have spent time on design, there's often no point estimating the total project.

commandlinefan 7y ago

> break down projects into the smallest deliverable "chunks" that you can Step 1: Figure out how to do this Step 2: Do it

mceachen 7y ago
Step 3: Discover that what you did was wrong (bad implementation or not what they wanted) Step 4: Go to step 1
alexhutcheson 7y ago
"Figure out how to do this" (aka create a high-level design) is a task, and you should be treat it the same way as you treat implementing a "chunk": estimate the effort it takes, prioritize it, and then work on it. The only wrinkle here is that you should not try to estimate any "chunks" of the implementation until you've finished your initial high-level design - there's no point estimating when you don't know what the "chunks" are. For big systems you may need to iterate on this a couple times to design sub-components of the larger system and create prototypes for systems that are critical to the project but might not work, etc. Estimation is incremental - sometimes you need to spend some non-trivial effort gathering the information you need before you can estimate the relative effort in building a system.
- LoSboccacc 7y ago
  > Figure out how to do this is a task this is the crux of the issue. on average you can estimate six month of work if you spend two weeks chopping it into tiny bits. rarely management allow that to happen and developers are rushed in with aggressive randomness. these "developers are bad at estimating" pieces are mostly people that try to estimate effort from complexity which is the wrong approach a priory because complexity hides the project unknowns, and this methodology, along all other comparative methods, is just a Gaussian curve built out off gut feelings: sometimes converges to a realistic estimate for large enough values, but more often than not the center just shows the developer team bias.
  - xyzzy_plugh 7y ago

jredwards 7y ago

> All the research indicates that software developers are terrible at estimating time, but are fairly accurate at estimating relative effort. I'd be interested in reading the research you're referencing, if you have something specific in mind.

HeyLaughingBoy 7y ago
I don't have a specific online reference to cite, but the Software Engineering Institute has a lot of research in this area. Specifically, the PSP (Personal Software Process) suggests using LOC as a proxy for size. e.g., a Small task might be 50 LOC, 200 LOC is medium, 700 is large. Then you can take those T-Shirt sizes and compare them to tasks you have done in the past and use that historical data to predict completion times. I'm simplifying it somewhat (e.g., there are also Complexity inputs), but the process does work if the organization is disciplined enough to follow it.
- necovek 7y ago
  There is a huge difference between "research" and "here's another idea" — I would personally love to hear about some research backing any estimation methodology. FWIW, I "feel" the big issue is with loosely defined requirements which shift as development progresses.
  - HeyLaughingBoy 7y ago
    PSP/TSP isn't "here's another idea." It's a research-backed software development process. And yes, while unmanaged, changing requirements is a major problem, even with proper requirements management you still have the time estimation problem.
- lloeki 7y ago
  LOCs don’t work either for more situations than I can count in my experience. I’ve had 50ish LOC stuff that implemented a tricky algorithm that was core to a system and required careful design, implementation and testing, right next to 700ish LOC stuff that was super straightforward and basically just laid itself out.

crdrost 7y ago

It’s a much bigger problem than this, sadly. The fact that you are even saying “If you want it to continue to work, you also have to be very careful how you communicate...” is the problem. The problem is that the way we manage things causes engineers and management to be against each other in principle, so that management is pushing one way and engineering is pushing the other way. Your advice is kind of like marital counseling, “let’s agree to not label Tom the person as an inconsiderate asshole but just talk about how he makes you feel,” “OK, well Tom, I am frustrated by your daily behavior that I perceive as assholish and inconsiderate” and I am sure it helps a little but I mean if you think Tom’s feelings aren’t going to be hurt you’re not giving him much credit. In fact if we were being honest with our managers we would be scandalized. We base our estimates on something like an 80% chance of getting something done (which already improves on the 50% chance by roughly doubling our estimate) and then every engineer I know has some informal rule about doubling or tripling the estimate before they hand it off to their manager. Those managers often add their own 20%. Then their managers see that something is up so they slash 50%. More than half of any project—perhaps even three quarters or more—is estimated as safety-factor. You would expect that if we were getting it wrong, we would be getting it wrong the other way, delivering early. The fact that we are late so often is a scandal in its own right. Adding more time buffer is clearly not helping much, since we have such a surfeit of it already. We must have wasted that safety buffer; there is no other explanation for how it disappeared. And once you have set up these luxurious margins of safety it is already too late. Why is the engineer adding that extra factor 4x or whatever it is? It is because management is already at cross-purposes to engineering. That engineer is adding that safety because they do not feel safe without it. Any insufficiently safe ski-lift in active use will see hundreds of buggy ill-specified implementations of safety harnesses: I will go on this ride but I am taking some precautions. The starting assumption has got to be that it is OK to fail on the deadline. This cannot be because of hidden information but must come as a result of trust: “we are eating into the safety buffer, that is totally OK and that is what it is there for, but I want to do whatever I can now to get you out of meetings and approach-planning and to fill out your timesheets for you and heck, even plan an amazing weekend between you and your fiancee to alleviate relationship stresses—and take any other distractions and loads off of your plate so that I can get your single-pointed focus on this feature and we can finish this.” If management makes themselves that safety harness, then there is no cross-purpose.

crumpets 7y ago

>After doing this for a while, you can use these relative effort estimates to project completion dates for new projects. This doesn't work. The problem is that you get periods where the work is relatively easy overall so 5 points ends up being tasks that take 2 days. Then you pick up new difficult features and estimate everything relative to all of the new hard tasks and you end up with 5 points meaning 5 days. Sure you can gauge relative difficulty of tasks between each other, but in a regular developer's day-to-day life the range should really be 1-100 or so rather than making stuff fit into 12 points or less.

alexhutcheson 7y ago
1. Keep your baseline tasks you're comparing against consistent for as long as you can (as long as the team remembers them and agrees on the effort they required). Don't use a task from week 2 to estimate week 3, etc. 2. Don't use a short time interval or moving average to determine your points->time mapping. You're looking for the long-run average mapping, not a specific estimate for this developer today.
- amatecha 7y ago
  +1 , this guy knows what's up. Relative estimates work really well, from my experience!
lifeisstillgood 7y ago
i like the 1-100 idea. so often the same points come back (a world of 1 and 3 pointers) but really have we broken that down? I think we should always have reference tasks in front of us -
yesbabyyes 7y ago
I think it's common to use the Fibonacci sequence as the difference between each valid weight increases pretty fast.

simplecomplex 7y ago

> All the research indicates Which research would that be?

singron 7y ago

The relative effort estimation seems to be more-or-less accurate for small chunks. However, at the beginning of a project, you don't know how many small chunks there are. In a small project, you can reasonably plan and estimate most of the chunks ahead of time and (eventually) plot out on a burn down chart that shows roughly when the project will complete. However, in a large project, you can really only plan the beginning of the project this way. Eventually there are too many unknowns and you risk all the downsides of waterfall design. In my experience, a project like this has a burn down chart that goes UP over time (i.e. chunks are added faster than they are completed). This makes estimating the project impossible, especially if the chart is still concave-up (i.e. the rate at which it increases is increasing). If you estimate large chunks, then you are less likely to add them over time, but you also won't be as good at estimating them. I think this is a good method for determining if a project is closer to a 1 year project or a 10 year project though. E.g. if you have 10 large equal chunks in a project and it ends up taking you 6 months to complete the first 1, you probably aren't going to finish by the end of the year, and you should evaluate if you are happy spending 5 years on it before you spend any more time.

jrochkind1 7y ago
In the "agile" practices I believe it comes from, I think it was actually basically only intended to answer "How much will we probably get done in the next week" or two or three at most, never to estimate an entire 'project'. Matching your perception of where it works and doesn't. i think? Anyone confirm/deny?