13 ms·

> Estimates are one of the hardest parts of software development. And also fundamental. When I was directly estimating big software projects the key, for me, w

> Estimates are one of the hardest parts of software development. And also fundamental. When I was directly estimating big software projects the key, for me, was to trust developers recommendations but apply a different multiplier for each developer. Multipliers ranged from x1 to x3. Those rare devs with x1 were, of course, a blessing. And those with x3 were not necessarily bad; they were often the ones working on the really hard problems. Of course, it meant getting to know those developers and a prior (which we set at x2, for new starters). Individuals were remarkably consistent in terms of their actual performance; so a x1 developer would almost always be a x1; a x1.5 would almost always be x1.5.

ragebol 5y ago
How much learning time did you typically need to arrive at that multiplier?
- ColinHayhurst 5y ago
  At the end of two completed projects from each new starter
LegitShady 5y ago
The multiplier was the same way I dealt with estimating time budget for homework in university. I would estimate the amount of time I thought it should take and multiply by 3 and it would usually take between 2-3x at the end. It's a little harder when the work is more nebulous and I don't have to do it for others but it's all based on personal estimations of productivity as well as understanding of the problem. It's not easy to do.
killjoywashere 5y ago
A Jira plug-in to use some sort of Bayesian system to adjust developer priors at the task and sprint level. That would be amazing.
- msluyter 5y ago
  Yeah I've always felt like the missing part of the loop (in the context of pointing stories) is looking at point estimates and then tracking how accurate they were, _by developer_. E.g., I throw 2 points, but it turns out the story takes something along the lines of 13 points, then my estimation was pretty low and my personal "estimation multiplier" is increased. Subsequently, actual point estimations could take the estimation multiplier into effect. Of course, this is mostly a toy idea that's fun to think about, but seems pretty untenable given that it requires keeping track of time, which most developers hate. That, and there'd no doubt be incentive to game the system by padding estimates or other shenanigans.
  - killjoywashere 5y ago
    Just look at completion rates within sprint. How many points did they commit to, how many did they finish. 2 weeks is a fine unit of time.
- tel 5y ago
  FogBugz used to do something of that nature. I never used it, so I'm not sure how well it worked out ultimately.
crooked-v 5y ago
I'm reminded of the classic saying: "The first 95% of the work takes 95% of the time, and the last 5% of the work takes the other 95% of the time."
mason55 5y ago
One thing I’ve found to be helpful is to make it clear that I won’t be mad about a long estimate If that’s how long it’s going to take then that’s how long it’s going to take. I think there’s a group of people who are used to getting lots of pushback on their estimates and so they estimate low to avoid that conflict. The future conflict of things being late is not something they have to deal with right now and maybe they actually will get it done. The key is backing it up continually and really not getting mad about estimates that are longer than I’d like. And, for people who I do think are sandbagging, asking more specific questions about the details of the estimate in a non-combative way. And of course some people really are bad at estimating. But I find if they don’t improve with coaching then it’s a symptom of a bigger problem (not thinking things through all the way) which manifests in other ways beyond estimation (eg poor design)
- commandlinefan 5y ago
  > I won’t be mad about a long estimate So... my experience with this mindset is that you won't be mad, but you'll just say "that's too long/expensive" and cancel the project entirely. Then the same project will come up again in two months with different wording, again and again, until I give you the estimate you think you can afford. And then the thing will end up taking longer than the original "too expensive" estimate (in part because too many things were rushed in the beginning to try to meet "the date"), but nobody will ever compare the final outcome against the original estimate anyway... because estimates are never meaningful.
  - stingraycharles 5y ago
    I think this attitude is too fatalistic. A good approach would be to see if there’s some way to break up the feature in smaller pieces, into smaller milestones, etc. If you have a manager that is unable to deal with this kind of stuff, then your problem is the manager, not the estimates. Estimates are extremely useful, and you’re doing yourself a disfavor if you think that they are never meaningful.
    - commandlinefan 5y ago
andruby 5y ago
Do you mind explaining why you say estimating is fundamental? I don’t disagree that it is important, I probably wouldn’t label it as fundamental, at least in a context of a company developing its own software internally.
- ColinHayhurst 5y ago
  Fair enough. It was the word that came to mind and I might have used a better one. Important doesn't quite capture it. Time is invariably the most precious resource, so perhaps "crucial".
- x0x0 5y ago
  It's one of the top two or three most important things. All teams have far more work than they can possibly do. A handful of things win; most things lose. Reasonable cost tradeoffs to pick which two or three things per quarter get worked on require estimates. And even in small startups, millions of dollars are spent on salaries per those decisions.
- majormajor 5y ago
  Even when developing your own stuff internally, you have a limited resource: your dev team's capacity to do work. The most important question for your internal software team is still "what should we work on" and estimates are a fundamental part of answering that. It's someone's job to figure out what things that team could do to provide the highest value. Usually some combo of product/sales/marketing/tech. But that's only half the equation! You also want to know how much it's going to cost, in terms of those scarce dev resources. If one project would add X value but take a year, and another set of projects would add Y, Z, and A value, but each only take four months, you want to compare Y+Z+A vs X to see what gets you the most value over the next year. So the better the estimates, the more rational you can be about what work gets done. If you have no estimates, you might pick a project that will go on for way too long, and not get the value out of it that you expected. (However, I doubt anywhere truly uses NO estimates. I've never seen a place that didn't at least have an informal level like "oh that'll be really hard" or "that'll take us a long time.")
  - 5y ago
satyrnein 5y ago
Joel Spolsky's team actually built that approach into a feature: https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
- HillRat 5y ago
  Fogbugz really was (is!) an underappreciated product. Jira’s become a necessary evil, but I really wish FB had managed to make a bigger dent in the market.
  - phist_mcgee 5y ago
    Something about the letter 'z' always screams out to me as a forgotten marker of 90s-00s culture. Really, any substitution for the letter 's' these days feels almost _antiquated_.
eikenberry 5y ago
Did you ever come across a person who was < x1, like a x0.5 (they overestimated)? I know developers generally undershoot, which is why we joke about making your best estimate and multiplying it by 2-3. But does no one really ever reliably overestimates? If the answer is never or rarely then that lends a reasonable amount of credence to the idea of just multiplying your estimates. Because you most likely should and the chance of overestimating is low.
- ColinHayhurst 5y ago
  Not in my experience
ChrisMarshallNY 5y ago
Good management advice. I was a first-line manager for a Japanese company (hard estimates, with lots of international frowny-faces, if the estimate was missed). That was pretty much what I did. We usually came relatively close to our estimates, but would sometimes encounter anomalous situations that would force us to re-evaluate. I found that my managers were pretty good at accepting these, as long as they were not too frequent, and as long as I didn't do the "Lucy in the Chocolate Factory" thing. They set hard plans, and we were not to deviate from them. But our customers often did not like the software that we delivered on-time, and to-spec. Since we were a hardware company, it was considered OK, but we hated it. So we delivered yesterday's technology; tomorrow. :( But I also ran a small team of really high-functioning C++ engineers. They stayed with me for decades, and I got to know them quite well. I can't even imagine running one of these shops with half a million inexperienced engineers that flow in and out like guppies in a pond. My management experience would be worthless for that environment. Since leaving, I have taken a very different tack with the software that I write. I assume that my estimates are basically fiction. I am going to make big changes, to suit changes in the deployment environment, customer expectations, competitive landscape, platform tech, etc. I tend to write very flexible software, in layers and modules. It allows me to react to these changes fairly quickly, and to maintain my very high quality standards, throughout. I often toss out huge gobs of code, when I hit one of these challenges. That actually makes me happy. The code I don't write, is the best code of all. Flexible software is a very double-edged sword. Most quality processes advise against it; for good reason. But I tend to know what I'm doing. I've been at this a long time, and have been very humbled by many, many mistakes. I would probably have a difficult time trusting developers with lesser experience to take the kinds of chances that I take. So my personal process depends heavily on me being me. I know myself, fairly well; warts and all. This means that I can trust the Principal engineer on my projects. I manage scope. My projects are small-batch, artisanal, projects. I don't bite off more than I can chew. That said, my "small batches" are bigger than a lot of folks' personal projects. I often spend months, writing my code, so change comes almost invariably, during my projects. Works for me. YMMV.
- xcambar 5y ago
  Honest question: how do you function in a team? I know a bunch of very self-aware devs, but they almost all have a very double-edged relationship with team work: theylove to learn and share yet see other people as impediments or risks on their way to delivery. How to improve on these situations?
twelvechairs 5y ago
Estimation depends on the unknowns which you can typically put down to overall task time and complexity of the task. If its is something they have done before and a short task (hours), x1 is fine. If it is something new and long (weeks), x5 may be underselling it. Breaking down a long task to be as granular as possible then rating each component against a scale of how much experience/confidence you have is the best approach to managing risk. As a manager, if you aren't asking people to explain this, you should be. I've never seen anyone do it (I also haven't worked anywhere massive), but monte-carlo analysis with a tornado chart etc. would be the best way of managing this kind of risk at a complex-systems scale. It is a system used by the top real-estate developers.
ska 5y ago
Individual factors are useful, but I've also found it very effective to ask for more that one confidence level, e.g. tell me how long you think it will take with 95% confidence, and with 50%. The spread on these is informative.
bsder 5y ago
I find that if you take the estimates and roll them up statistically, you get a pretty good result. The problem is that everybody schedules to the estimate assuming it's the 90% point (project done) when it's actually a pretty accurate mean (50% point). In reality, the number is probably closer to the 33% probability because software can be an unbounded amount of time late but only a bounded amount of time early. The primary problem is that when you feed people the 90% number up front they freak out. The secondary problem is that nobody goes back after the project was done and checks whether the 90% estimate was accurate (it generally is pretty close).
YZF 5y ago
I don't trust multipliers. I often find those rules of thumb to be very poor predictors. Since if you don't know how can you predict some fixed fraction of the work? I led a small team to deliver software over a multi-year project. The software was used to control a machine. The machine was contracted as part of a multi-million dollar deal. We delivered software on time and with good quality. It was very far from trivial. The key to this is that the team had experience in doing similar projects, so we generally knew how long things take. We knew what needed to get done. We knew we would be debugging the machines. We knew we would need to work around issues. We knew we'd be limited until the first prototypes are assembled. We knew what existing tools and libraries we could reuse. And we also knew what's the minimal viable product and what are nice to haves. For me personally this was the third consecutive project of building very similar machine control software, for very similar applications, using very similar technologies. This wasn't exactly the same but I went through this process and I knew how it works. The first of those actually failed because the software wasn't ready on time (and other reasons, but that was the main one). A team with no experience, building something new, has zero chance of correctly estimating anything. Not only will they likely not deliver on time, they might not deliver at all, ever. Or they'll deliver something that doesn't work. Someone external who has experience with these teams might be able to provide the right estimate (e.g. they'll say it'll take them 3 years and they'll deliver nothing). In my current job I asked a junior engineer for an estimate, he broke down stuff to tasks, and came back with something like "a month". Ten months later it wasn't done. Much simpler technically than the other project but the engineer didn't really see all the details, didn't have experience with similar projects, so he basically just made stuff up. You always have to plan for surprises and you need to have contingencies. An estimate is always some sort of distribution but for a large enough project these distributions do form some sort of coherent picture. Sure, you might have a surprise, but you do the risky things first to reduce those risks.
- mtippett 5y ago
  Yay #noestimatemultipliers
mtippett 5y ago
Arggh. I hate multipliers. It's not poor estimates that need to be doubled. It's the lack of experience in identifying the complexity or likely risks in a project. So rather than multiply, ask for risks. You'll find that most engineers know the risks (or at least they may know it is without risk). As you explore risks, you'll find the estimates will lengthen automatically. So rather than do the multiplier, explore the risks. They are what makes projects late, not intrinsic engineer productivity. You'll end up with better engineers who know themselves and their problem spaces better. You'll also get a clearer view of what could screw up your project.
- nudgeee 5y ago
  I agree that project risks should be explored and multipliers are not substitutes for them. However multipliers are useful for ongoing dynamic distractions outside of the project that reduce your productivity, for example company meetings, hiring/debriefs, getting pulled into critical bugs, helping to support colleagues, etc.
- feoren 5y ago
  I don't think OP is saying one engineer is 3x "more productive"; rather they're saying that engineers tend to be consistent in how much they over- or under-estimate tasks. It sounds a little like "story points", where each developer may end up having their own estimation scale.

ragebol 5y ago

How much learning time did you typically need to arrive at that multiplier?

ColinHayhurst 5y ago
At the end of two completed projects from each new starter

LegitShady 5y ago

The multiplier was the same way I dealt with estimating time budget for homework in university. I would estimate the amount of time I thought it should take and multiply by 3 and it would usually take between 2-3x at the end. It's a little harder when the work is more nebulous and I don't have to do it for others but it's all based on personal estimations of productivity as well as understanding of the problem. It's not easy to do.

killjoywashere 5y ago

A Jira plug-in to use some sort of Bayesian system to adjust developer priors at the task and sprint level. That would be amazing.

msluyter 5y ago
Yeah I've always felt like the missing part of the loop (in the context of pointing stories) is looking at point estimates and then tracking how accurate they were, _by developer_. E.g., I throw 2 points, but it turns out the story takes something along the lines of 13 points, then my estimation was pretty low and my personal "estimation multiplier" is increased. Subsequently, actual point estimations could take the estimation multiplier into effect. Of course, this is mostly a toy idea that's fun to think about, but seems pretty untenable given that it requires keeping track of time, which most developers hate. That, and there'd no doubt be incentive to game the system by padding estimates or other shenanigans.
- killjoywashere 5y ago
  Just look at completion rates within sprint. How many points did they commit to, how many did they finish. 2 weeks is a fine unit of time.
tel 5y ago
FogBugz used to do something of that nature. I never used it, so I'm not sure how well it worked out ultimately.

crooked-v 5y ago

I'm reminded of the classic saying: "The first 95% of the work takes 95% of the time, and the last 5% of the work takes the other 95% of the time."

mason55 5y ago

One thing I’ve found to be helpful is to make it clear that I won’t be mad about a long estimate If that’s how long it’s going to take then that’s how long it’s going to take. I think there’s a group of people who are used to getting lots of pushback on their estimates and so they estimate low to avoid that conflict. The future conflict of things being late is not something they have to deal with right now and maybe they actually will get it done. The key is backing it up continually and really not getting mad about estimates that are longer than I’d like. And, for people who I do think are sandbagging, asking more specific questions about the details of the estimate in a non-combative way. And of course some people really are bad at estimating. But I find if they don’t improve with coaching then it’s a symptom of a bigger problem (not thinking things through all the way) which manifests in other ways beyond estimation (eg poor design)

commandlinefan 5y ago
> I won’t be mad about a long estimate So... my experience with this mindset is that you won't be mad, but you'll just say "that's too long/expensive" and cancel the project entirely. Then the same project will come up again in two months with different wording, again and again, until I give you the estimate you think you can afford. And then the thing will end up taking longer than the original "too expensive" estimate (in part because too many things were rushed in the beginning to try to meet "the date"), but nobody will ever compare the final outcome against the original estimate anyway... because estimates are never meaningful.
- stingraycharles 5y ago
  I think this attitude is too fatalistic. A good approach would be to see if there’s some way to break up the feature in smaller pieces, into smaller milestones, etc. If you have a manager that is unable to deal with this kind of stuff, then your problem is the manager, not the estimates. Estimates are extremely useful, and you’re doing yourself a disfavor if you think that they are never meaningful.
  - commandlinefan 5y ago

andruby 5y ago

Do you mind explaining why you say estimating is fundamental? I don’t disagree that it is important, I probably wouldn’t label it as fundamental, at least in a context of a company developing its own software internally.

ColinHayhurst 5y ago
Fair enough. It was the word that came to mind and I might have used a better one. Important doesn't quite capture it. Time is invariably the most precious resource, so perhaps "crucial".
x0x0 5y ago
It's one of the top two or three most important things. All teams have far more work than they can possibly do. A handful of things win; most things lose. Reasonable cost tradeoffs to pick which two or three things per quarter get worked on require estimates. And even in small startups, millions of dollars are spent on salaries per those decisions.
majormajor 5y ago
Even when developing your own stuff internally, you have a limited resource: your dev team's capacity to do work. The most important question for your internal software team is still "what should we work on" and estimates are a fundamental part of answering that. It's someone's job to figure out what things that team could do to provide the highest value. Usually some combo of product/sales/marketing/tech. But that's only half the equation! You also want to know how much it's going to cost, in terms of those scarce dev resources. If one project would add X value but take a year, and another set of projects would add Y, Z, and A value, but each only take four months, you want to compare Y+Z+A vs X to see what gets you the most value over the next year. So the better the estimates, the more rational you can be about what work gets done. If you have no estimates, you might pick a project that will go on for way too long, and not get the value out of it that you expected. (However, I doubt anywhere truly uses NO estimates. I've never seen a place that didn't at least have an informal level like "oh that'll be really hard" or "that'll take us a long time.")
- 5y ago

satyrnein 5y ago

Joel Spolsky's team actually built that approach into a feature: https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...

HillRat 5y ago
Fogbugz really was (is!) an underappreciated product. Jira’s become a necessary evil, but I really wish FB had managed to make a bigger dent in the market.
- phist_mcgee 5y ago
  Something about the letter 'z' always screams out to me as a forgotten marker of 90s-00s culture. Really, any substitution for the letter 's' these days feels almost _antiquated_.

eikenberry 5y ago

Did you ever come across a person who was < x1, like a x0.5 (they overestimated)? I know developers generally undershoot, which is why we joke about making your best estimate and multiplying it by 2-3. But does no one really ever reliably overestimates? If the answer is never or rarely then that lends a reasonable amount of credence to the idea of just multiplying your estimates. Because you most likely should and the chance of overestimating is low.

ColinHayhurst 5y ago
Not in my experience

ChrisMarshallNY 5y ago

Good management advice. I was a first-line manager for a Japanese company (hard estimates, with lots of international frowny-faces, if the estimate was missed). That was pretty much what I did. We usually came relatively close to our estimates, but would sometimes encounter anomalous situations that would force us to re-evaluate. I found that my managers were pretty good at accepting these, as long as they were not too frequent, and as long as I didn't do the "Lucy in the Chocolate Factory" thing. They set hard plans, and we were not to deviate from them. But our customers often did not like the software that we delivered on-time, and to-spec. Since we were a hardware company, it was considered OK, but we hated it. So we delivered yesterday's technology; tomorrow. :( But I also ran a small team of really high-functioning C++ engineers. They stayed with me for decades, and I got to know them quite well. I can't even imagine running one of these shops with half a million inexperienced engineers that flow in and out like guppies in a pond. My management experience would be worthless for that environment. Since leaving, I have taken a very different tack with the software that I write. I assume that my estimates are basically fiction. I am going to make big changes, to suit changes in the deployment environment, customer expectations, competitive landscape, platform tech, etc. I tend to write very flexible software, in layers and modules. It allows me to react to these changes fairly quickly, and to maintain my very high quality standards, throughout. I often toss out huge gobs of code, when I hit one of these challenges. That actually makes me happy. The code I don't write, is the best code of all. Flexible software is a very double-edged sword. Most quality processes advise against it; for good reason. But I tend to know what I'm doing. I've been at this a long time, and have been very humbled by many, many mistakes. I would probably have a difficult time trusting developers with lesser experience to take the kinds of chances that I take. So my personal process depends heavily on me being me. I know myself, fairly well; warts and all. This means that I can trust the Principal engineer on my projects. I manage scope. My projects are small-batch, artisanal, projects. I don't bite off more than I can chew. That said, my "small batches" are bigger than a lot of folks' personal projects. I often spend months, writing my code, so change comes almost invariably, during my projects. Works for me. YMMV.

xcambar 5y ago
Honest question: how do you function in a team? I know a bunch of very self-aware devs, but they almost all have a very double-edged relationship with team work: theylove to learn and share yet see other people as impediments or risks on their way to delivery. How to improve on these situations?

twelvechairs 5y ago

Estimation depends on the unknowns which you can typically put down to overall task time and complexity of the task. If its is something they have done before and a short task (hours), x1 is fine. If it is something new and long (weeks), x5 may be underselling it. Breaking down a long task to be as granular as possible then rating each component against a scale of how much experience/confidence you have is the best approach to managing risk. As a manager, if you aren't asking people to explain this, you should be. I've never seen anyone do it (I also haven't worked anywhere massive), but monte-carlo analysis with a tornado chart etc. would be the best way of managing this kind of risk at a complex-systems scale. It is a system used by the top real-estate developers.

ska 5y ago

Individual factors are useful, but I've also found it very effective to ask for more that one confidence level, e.g. tell me how long you think it will take with 95% confidence, and with 50%. The spread on these is informative.

bsder 5y ago

I find that if you take the estimates and roll them up statistically, you get a pretty good result. The problem is that everybody schedules to the estimate assuming it's the 90% point (project done) when it's actually a pretty accurate mean (50% point). In reality, the number is probably closer to the 33% probability because software can be an unbounded amount of time late but only a bounded amount of time early. The primary problem is that when you feed people the 90% number up front they freak out. The secondary problem is that nobody goes back after the project was done and checks whether the 90% estimate was accurate (it generally is pretty close).

YZF 5y ago

I don't trust multipliers. I often find those rules of thumb to be very poor predictors. Since if you don't know how can you predict some fixed fraction of the work? I led a small team to deliver software over a multi-year project. The software was used to control a machine. The machine was contracted as part of a multi-million dollar deal. We delivered software on time and with good quality. It was very far from trivial. The key to this is that the team had experience in doing similar projects, so we generally knew how long things take. We knew what needed to get done. We knew we would be debugging the machines. We knew we would need to work around issues. We knew we'd be limited until the first prototypes are assembled. We knew what existing tools and libraries we could reuse. And we also knew what's the minimal viable product and what are nice to haves. For me personally this was the third consecutive project of building very similar machine control software, for very similar applications, using very similar technologies. This wasn't exactly the same but I went through this process and I knew how it works. The first of those actually failed because the software wasn't ready on time (and other reasons, but that was the main one). A team with no experience, building something new, has zero chance of correctly estimating anything. Not only will they likely not deliver on time, they might not deliver at all, ever. Or they'll deliver something that doesn't work. Someone external who has experience with these teams might be able to provide the right estimate (e.g. they'll say it'll take them 3 years and they'll deliver nothing). In my current job I asked a junior engineer for an estimate, he broke down stuff to tasks, and came back with something like "a month". Ten months later it wasn't done. Much simpler technically than the other project but the engineer didn't really see all the details, didn't have experience with similar projects, so he basically just made stuff up. You always have to plan for surprises and you need to have contingencies. An estimate is always some sort of distribution but for a large enough project these distributions do form some sort of coherent picture. Sure, you might have a surprise, but you do the risky things first to reduce those risks.

mtippett 5y ago
Yay #noestimatemultipliers

mtippett 5y ago

Arggh. I hate multipliers. It's not poor estimates that need to be doubled. It's the lack of experience in identifying the complexity or likely risks in a project. So rather than multiply, ask for risks. You'll find that most engineers know the risks (or at least they may know it is without risk). As you explore risks, you'll find the estimates will lengthen automatically. So rather than do the multiplier, explore the risks. They are what makes projects late, not intrinsic engineer productivity. You'll end up with better engineers who know themselves and their problem spaces better. You'll also get a clearer view of what could screw up your project.

nudgeee 5y ago
I agree that project risks should be explored and multipliers are not substitutes for them. However multipliers are useful for ongoing dynamic distractions outside of the project that reduce your productivity, for example company meetings, hiring/debriefs, getting pulled into critical bugs, helping to support colleagues, etc.
feoren 5y ago
I don't think OP is saying one engineer is 3x "more productive"; rather they're saying that engineers tend to be consistent in how much they over- or under-estimate tasks. It sounds a little like "story points", where each developer may end up having their own estimation scale.