8 ms·
There is lots of good advice here, but I'm always amazed that these articles miss the #1 improvement to estimates recommended and validated by scientific studie
by ohthehugemanate 3y ago
There is lots of good advice here, but I'm always amazed that these articles miss the #1 improvement to estimates recommended and validated by scientific studies: do not estimate in units of time.
Everything else here applies, but don't estimate in time. Estimate in difficulty points, t shirt sizes, cups of coffee, gold stars, anything will do. Then as you track your outcomes (see TFA), measure the average relationship between your units and time. Then use that relationship to project a timeline based on your gold stars or whatever.
Using the law of large numbers to average externalities and developer inaccuracy into your time estimate has been the most accurate method since Kahneman and Tversky's Nobel prize winning studies in the 1970s.
- techdragon 3y agoI’m always trying to gather extra ammunition for when I’m arguing for points based estimation methodology… do you (or anyone else) have any favourite scientific studies on this ?
- switch007 3y agoMy previous manager and PM would scrutinise the time taken in Jira, then create a personal spreadsheet to roughly map points to hours/days (because senior stakeholders don't think in cups of coffee or t-shirt sizes), then refer to the spreadsheet privately during estimation. But he eventually let slip about the existence of it So we just bypassed the charade and starting giving hours/days. The whole team was really just doing hours/days -> points mapping in their heads during estimation anyway.
- ohthehugemanate 3y ago> The whole team was really just doing hours/days -> points mapping in their heads during estimation anyway. This is a common story. But because of the calculation step, your PM was still mapping estimated time units to actual time consumed, and using that mapping for predictions. So you were getting some of the benefits of an abstracted unit (accuracy, averaging in unexpected events) without the stress reduction or focus on quality over time. But if it worked for your team, I would advocate to keep it. Human groups are hard enough to coordinate as it is, there's no need to hew to a dogma.
- SanderNL 3y agoSo your team does basically, let’s say, 10 gold stars a week. Your epic and VeryNecessaryFeature(c) takes 30 gold stars. Pointy hair boss: this takes 3 weeks. (and make it 2) How is this different from skipping the proxy and just using time directly?
- tbrake 3y agoThe answer is not having a PHB. No non-unit-of-time estimation system will matter if there isn't buy-in from all levels. If the Directors/VPs/C-Suite etc view development in assembly line terms, then Xpoints = Ytime can never be avoided and yes, it just becomes an exercise in frustration. To date I haven't found a good antidote other than avoiding workplaces that view estimations as exactimations of time.
- Ensorceled 3y agoAh, so the best way to provide time estimates so that the rest of the business can do planning is to not work at a business that needs to do planning.
- tbrake 3y agoOn a story by story basis saying points=time is an idea as silly as your deliberate misinterpretation of my post, yes.
- konschubert 3y agoIt’s different because it can remove a bias out of estimations.
- SanderNL 3y agoBut eventually everyone knows that your gold stars stand for weeks. So how does this help?
- konschubert 3y ago
- OJFord 3y agoOk, so how many 'difficulty points, t shirt sizes, cups of coffee, gold stars, anything's fit in a two week sprint? I hate pretending that we're estimating 'complexity'.
- ohthehugemanate 3y agoHow many has your team completed on average in the past sprints? It's not pretense. WTF should an engineer care about time? An engineer has to care about Done Right. It takes as long as it takes. A PM cares about time, and they can looks at the past history of similar tasks to estimate it very accurately. The abstracted "effort" unit is a way to identify "similar tasks". It's also very well demonstrated that human brains suck at estimating time. Estimating difficulty is way more consistent. But if you hate thinking in terms of complexity you can do the same thing by making sure your tasks are evenly sized, and averaging task throughput over time. Or if you also hate your pm you could tag similar tasks and have them do the leg work without the math to help.
- OJFord 3y ago> How many has your team completed on average in the past sprints? In other words 'what is the average conversion from complexity to time'; in other words 'let us try to point time but call it complexity'. > WTF should an engineer care about time? An engineer has to care about Done Right. It takes as long as it takes. That's great, but then #points in a fixed-time sprint should be expected to be all over the place.
- kqr 3y agoI'm a bit surprised by the "validated by scientific studies" bit. If there's anything I got out of the likes of Kahneman (Noise), Hubbard (How To Measure Anything), and Tetlock (no particular reference) it's the complete opposite: If you estimate in t-shirt sizes people will come to false agreements. The variation in what people mean when they give the same t-shirt size is enormous. Even the same person estimating similar tasks weeks apart may give the same t-shirt size but when asked how to translate it to time will give wildly varying numbers. The only thing that actually stands up to hindsight verification is estimating in as concrete units as possible – ideally calendar dates. Of course, if you invoke statistical properties of past estimations to convert t-shirt sizes to calendar dates, you're sort of doing that already. But why do it behind people's backs? In my experience (and drawing from system theory) it is much more useful to allow people to receive the feedback of their successes and failures.
- ohthehugemanate 3y agoI'm so happy to see someone who has actually read some of these studies on here. There are only 2 reasons to do the translation behind peoples' backs AFAIK: 1) also from kahneman et al we know that humans are bad at estimating time, and much more consistent and accurate about measuring effort. And consistency is required for the law of large numbers to apply. By estimating in consistent non time units you avoid those cognitive biases. 2) humans will very quickly start to unconsciously bias the units if they feel pressure. Speaking in terms of time puts one very close to a lot of pressure points for a lot of people. Ideally engineers are concerned with doing things Right, and PMs are using past performance to predict how long that will take.
- badbanana 3y agoCustomer: What will this project cost? You: Five XL T-shirts. Excuse the snark, but a lot of business decisions are made based on how much Project X will cost. You likely get paid based on time, so it's quite natural to gravitate toward time units - any intermediary unit is just an abstraction.
- midasz 3y agoPeople sell projects for a fixed price? How does that work? You either overpay a ton of underpay right? And how would you even say what a project will cost? Projects change all the time, and there is no way to know the entire scope beforehand unless it's really really small. Otherwise you're just lying. Genuinely how does this work?
- another-dave 3y agoBut you do this in the non-software world all the time — if you leave your car in for a service, you'll get charged £X base on it taking them Y amount of time. Some cars will take them ±10% but it balances out in the end. Or if you're getting building work on your house, they'll caveat that it'll cost you £X, saving any unforeseen circumstances that arise (if you've asbestos in your attic, they're not going to stick to the original price and just eat the cost) In consulting, most clients want a fixed cost for a fixed scope, with any deviations coming as change requests & billed accordingly. Generally people are hestitant to agree to an agile model unless you've already got a lot of trust built up.
- runlaszlorun 3y agoPersonally, I’ve never seen one go well though all of the projects have been relatively complex with no repeatability. The vendor ends up losing money and the client gets a deal on paper but not in reality when you realize the corners that were cut to shove a product at the door. You could generally go back and look at the emails on the project and see the exact point in time (often about 2/3 in) were the vendor realized they were losing money and started cutting whatever possible to not lose their shirt on the project. I suppose that if projects were relatively straightforward like “we’re building you a wordpress site with X number of templates” one may have a chance at estimating upfront- but I’ve never worked on projects that predictability.
- regularfry 3y agoThe law of large numbers doesn't help here because a) the error distribution is strongly one-sided (it being much, much easier for something to take twice as long than half as long) and b) there aren't enough projects, typically, for implementation teams to stay stable for long enough for the central limit theorem to compensate. I wouldn't be at all surprised if the relationship between estimate and actual typically followed a power law relationship, in which case speaking about any averaging over time is meaningless anyway.
- MakeUsersWant 3y agoI've heard it's lognormal, which doesn't follow the central limit theorem either.
- ohthehugemanate 3y agoGreat response. Part of the point of estimating difficulty is exactly to reduce the one-sidedness evident in time estimations. Human brains are much more accurate and consistent with effort estimation than time. It's still very likely a beta distribution but for the level of accuracy we're talking about it's just fine. We should probably clarify the "bar" here: the very best time estimators never beat about +30% delta from actual time. (Note that I'm not using +/- intervals because it's only ever +). So how many points (effort/time ratio values) does it take for convergence to get your effort/time ratio mean within 30% of the peak of the curve?
- regularfry 3y agoI don't think that 30% bar is commonly understood. From details in conversations I've had in the past where the reaction to (for instance) a 50% miss was "we can't ever be that bad at estimating again", there's a pervasive attitude that it's a fault in implementation teams that they're not accurate to the day. There's no middle ground: a team that can't give "good enough" estimates is held as incompetent, while the best anyone can ever do would be unacceptable. What's interesting on that specific 50% miss I saw is that the original estimate from the actual teams themselves was argued down as "too big, do it again" by their own management...
- sanitycheck 3y agoYet whenever I've seen this attempted in practice it's failed miserably. It sounded good when I first heard it, but now I treat it as cargo cult nonsense. Maybe it works but nobody does it right? A point is a different size for everyone, but a day is a universal (ok technically global) unit. We deal in time all the time, we know how long an hour is, a day, a week. We can remember "I did something like this before, it took me 2 weeks", not "it took me 13 points". No manager can predictably translate story points into days, which they need in order to pitch to customers and manage their budgets. The first thing I ask when I have to estimate in SP is "how many hours is 1SP?" and after a few minutes of the usual back-and-forth, whoever has to actually use these damn estimates always says something like "I treat 1SP as half a day". Bingo, now I can give you a number you can use.
- vp8989 3y agoIt's absolutely cargo cult nonsense and judging by the surrounding comments I'm SO glad we've all finally decided to stop pretending like it isn't.
- midasz 3y ago> We deal in time all the time, we know how long an hour is, a day, a week. We can remember "I did something like this before, it took me 2 weeks", not "it took me 13 points". I really don't get this. Can you guys genuinely work 2 weeks on a feature without doing anything else?. No extra meetings, no dependencies, no incidents, no coworkers to help? Something simple can take me 2 weeks or 2 days - it really depends on all the other stuff going on when I'm working. That's why imo it's so silly to estimate time unless you're tracking every minute you work on that specific thing and manage to seal yourself in a box away from distractions. The thing I remember when estimating is. Huh, last time I touched this feature it went really smooth. Doesn't look like much work, adding a validator here and I've got an example right there. Factor in some testwork and I guess it's like 3 points? > No manager can predictably translate story points into days, which they need in order to pitch to customers and manage their budgets. Either you're a godlike estimator or you disappoint your manager A LOT. Who the hell are these people that can reliably say feature X will take me 8 weeks? Is that also not just an estimate? I'd rather wallow in the vagueness of points than get my ass reprimanded because I said something would take X weeks. If you're looking for absolute predictability you need factory line work.
- alkonaut 3y agoEstimating in something other than hours still needs to be converted for hours in order to be useful (for resource allocation, budgeting, etc). The only thing you can do with "points" is compare things. Not unimportant, but still not enough. In "estimating in the small" such as for a development iteration one can often get away without a precise conversion to hours and dollars. But in deciding whether to do ProjectA or ProjectB each being 100+ man-year projects, or whether to reassign the new hire to the team of ProjectA or ProjectB, you still need hours. It's not going to help to say the project is six hundred thousand points, unless you can convert it to the real estimate which is "a 20-40 man team for 4-8 years"
- ohthehugemanate 3y ago> Estimating in something other than hours still needs to be converted for hours in order to be useful Yes. Did you see the part of my comment where I explained how you do that? The fact that this is the most accurate estimation method is well established in research. You can get close with some other methods, like tri-partite estimation, but they're more cumbersome. WHY it helps so much to abstract away from direct time estimation, has a few going theories. One obvious reason is that the abstraction and tracking the relationship to real time brings into play the Law Of Large Numbers, the same mathematical law that allows casinos to profit on games of pure chance. The more sprints you have under your belt, the more your conversion rate averages in unexpected external influences like illness, windows updates, upstream problems, etc, in proportion to their actual occurrence rates. Another of the most popular theories is that human brains have specialized areas/heuristics for predicting time vs predicting effort, which come with different cognitive biases. Fundamentally I can't give you a definitive answer to why, only that speaking in terms of difficulty with engineers, and converting to time to talk to business planning, is well demonstrated to greatly improve accuracy in controlled studies. In fact, the best estimators, when predicting time, still predict 130% of the time needed for a complex task. Using abstracted measures it is not unusual to see accuracy within 5% or 10% of actual.
- disgruntledphd2 3y ago> The fact that this is the most accurate estimation method is well established in research. You can get close with some other methods, like tri-partite estimation, but they're more cumbersome. You keep talking about the research, is there any chance you can provide some references? I'd like to review this research as what you say sounds plausible and I'd like to know more.
- ochronus 3y agoI've seen 3 major uses for estimations (besides the obvious forcing function to align on complexity/ideas among the team members): 1. Somebody really needs a time-based estimate for the project, e.g. the customer wants feature X by a certain time and we need to figure out if it's feasible or not in that timeframe, or there's a certain external event (expo, or any other kind of marketing event) with a fixed date and we need to see if we can make it. This is usually rare in traditional product companies. 2. We need to see if the project is worth it, in general. Maybe it's just too much effort to even consider. 3. We need to prioritize a couple of projects/opportunities. ICE/RICE comes into play, and we need to stack rank based on return on investment. Relative effort in meaningless units (tshirt, cups, etc.) make perfect sense here.
- time0ut 3y agoThe reality (mine at least) is that every time I've been asked to estimate in t-shirt size or points, there is a helpful chart mapping those to units of time.