18 ms·

Software effort estimation is mostly fake research (2021)

326 points by greghn 3y ago

yacine_ 3y ago
so true
aetherson 3y ago
Site seems hugged to death.
Espressosaurus 3y ago
Site is unresponsive. Internet Archive link: https://web.archive.org/web/20230615230250/https://shape-of-code.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://web.archive.org/web/20230615230250/https://shape-of-...
29athrowaway 3y ago
Even this has more merit than COCOMO https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...
satisfice 3y ago
The datasets are not even data in the first place. If you want to understand estimations and projects, you have to study them. This is a social science project. It's not a math game. It's not a problem that you can solve by collecting oversimplified numbers from dubious sources based on incoherent definitions of the units of measurement. If somebody reports that they spent x days developing a component, what does that mean? What were they doing during those days? How do we know what they were doing? We don't know unless a researcher was right there in the room taking notes in real-time. There are so many confounding variables involved, even if your data collection is perfect, that analysis of the data is a process fraught with opportunities for accidental overfitting or researcher bias.
- readthenotes1 3y ago
  The one bit of project estimation research that I actually believed was that project estimates are closely affected by timeline desires of major finders/"stakeholders". Anecdata: I once asked a VP of a major telecom why the project deadline was Sept 30 and not Oct 1 or Sept 28, and he replied "because my quarterly bonus depends on it"
awb 3y ago
Ran a successful digital agency for years. Estimation was mostly guesswork. There’s a theory that work expands to fill the time you give it and I found that to be largely true. Other agencies would quote 3 months for projects we’d complete in 6 weeks. Even then I was taking engineers 3 week estimates and doubling them for buffer. We didn’t do anything special other than commit individuals to a single project, or two at most. We would of course add additional resources if necessary to hit deadlines and then adjust future estimates as needed. After a few years estimation felt more instinctual than scientific.
- michelpp 3y ago
  > There’s a theory that work expands to fill the time you give it https://en.m.wikipedia.org/wiki/Parkinson's_law https://en.m.wikipedia.org/wiki/Parkinson's_law
  - Solvency 3y ago
    It can also just barely squeeze into the time you give it, by the skin of your teeth.
    - someweirdperson 3y ago
      The trick is to avoid that people who are doing the work are aware of the estimates or deadlines.
- evantbyrne 3y ago
  I suspect that experience is a major factor as well. You somehow knew to double those estimates. At the agencies I've worked at over the years, less experienced individuals seemed to consistently underestimate the time it would take to do things and then complete those tasks haphazardly/late/bugged. I went freelance and haven't had any trouble providing accurate estimations.
  - mgkimsal 3y ago
    I would guess a repeatable/known type of service or project helps a lot as well. Most of my engagements are relatively different from each other, and I'm doing different things all the time. I've worked places which were offering a narrower set of functionality, with limited customizations, and that's generally much easier to estimate. Also much easier to have estimates in which you've said 'no' to things altogether, vs spending time researching/testing/trialing stuff that's new to you.
quickthrower2 3y ago
My instinct is that if you can put numbers and predictions based on real world estimations, then you are estimating simple things. For example, you can estimate how long it will drive from A to B. Google maps does a brilliant job of this, showing the busy sections. Often I get there +/- a minute from what is estimated over a 60 minute journey. But estimating how long someone will take to do a project, where the project by definition is something new in the context of the people doing it. If it wasn't then the code would already exist, and the estimate will be "how long to deploy that thing you already have using CI". So in software you are always doing something new. We are building Sydney Opera Houses, not prefab housing estates.
- x86x87 3y ago
  Also, to get better estimates you need to sometimes do some [research] work. Without it ykur guesses are even wilder but with it you have to ack that you've spent time on it, evenif you never get to do the actual work.
  - ethbr0 3y ago
    I.e. spiral https://en.m.wikipedia.org/wiki/Spiral_model https://en.m.wikipedia.org/wiki/Spiral_model
    - x86x87 3y ago
      TIL. Thanks for sharing this
      - ethbr0 3y ago
        Sure! I tossed some thoughts on the sibling comment above yours.
    - quickthrower2 3y ago
      That is interesting. Going to look more into this. When something looks like common sense codified, then I like it :-)
mechanicker 3y ago
Reminds me of my early days as a rookie people manager. I was asked to estimate a project for a team that I had no idea about their abilities. I was very hesitant to estimate since I had no data to begin with. My manager asked me to throw a number and see if there is a push back!!! I soon made a decision not to be a manager responsible for guess work based estimation and moved back to an IC role where I estimate effort for work assigned to me.
ElFitz 3y ago
Whenever the topic of software estimates comes up, I am reminded of Erik Bernhardsson’s piece[1]. It was a great read, and still proves useful when trying to explain how things can sometimes go quite wrong. [1]: https://erikbern.com/2019/04/15/why-software-projects-take-longer-than-you-think-a-statistical-model.html https://erikbern.com/2019/04/15/why-software-projects-take-l... Edit: it has come up a few times on hn. See https://news.ycombinator.com/item?id=19671673 https://news.ycombinator.com/item?id=19671673
- e12e 3y ago
  This is great, thank you. Another approach is to do forecasting based on historical throughput (eg tasks/day). Generally using "story points" or other guesstimates turn out to be redundant if you have actual data (actual project by mostly the same team). It takes into account stuff like n% of issues "blowing up" into subtasks, and some issues taking longer. I read a great article on the subject which I've unfortunately misplaced - but the general gist is touched on in these: "Forecasting software project’s completion date through Monte Carlo Simulation" https://towardsdatascience.com/forecasting-software-projects-completion-date-through-monte-carlo-simulation-c1baa5bcf976 https://towardsdatascience.com/forecasting-software-projects... "When will it be done? A data-driven approach to estimating project completion" https://screenful.com/blog/when-will-it-be-done-a-data-driven-approach-to-estimating-project-completion https://screenful.com/blog/when-will-it-be-done-a-data-drive... "When Will It Be Done: Project Forecasting in Kanban(ize)" (product ad) https://kanbanize.com/blog/project-forecasting/ https://kanbanize.com/blog/project-forecasting/ "Project Forecaster" https://github.com/rodrigozr/ProjectForecaster/wiki https://github.com/rodrigozr/ProjectForecaster/wiki Ed: forgot this one linked from forecaster above: ”#NoEstimates Project Planning Using Monte Carlo Simulation" https://www.infoq.com/articles/noestimates-monte-carlo/ https://www.infoq.com/articles/noestimates-monte-carlo/ And maybe this project is of interest: https://microsoft.github.io/forecasting/ https://microsoft.github.io/forecasting/ In general I'd like to be able to just point a tool at N "completed" GitHub projects, and a fresh one - and get a forecast that adjusts as issues are completed and added (possibly filtering on issue tags in completed and new project) - but I'm not aware of anything out of the box for that.
  - 3y ago
pyrolistical 3y ago
Lost me at the first sentence > Effort estimation is an important component of any project, software or otherwise. The most “important component”, what does that even mean? Do they mean task? Isn’t the most important thing to deliver results? Which could mean working software or aborting bad projects
- am44jnsf 3y ago
  it says "an important", not "most important". It's correct, it is an important.
MuffinFlavored 3y ago
If your manager says "I need an estimate" and you can't tell your manager "no, that's fake research/not worth my time", does it really matter what is/isn't "fake research"?
- scraptor 3y ago
  Does your manager expect you to obtain your estimate using a machine learning approach from a research paper?
neodypsis 3y ago
> Agile techniques are widely used these days, and datasets from the 1990s are not applicable. What datasets do researchers use to build Agile effort estimation models? In an undergraduate software engineering course I took we learned an effort estimation methodology that assumed a particular way of working (similar to Agile), but also a particular software architecture. The problem was that students then had to adopt such an architecture, even though it wasn't the best fit to the problem at hand, just so that they could complete the effort estimate. Are there methods that are independent of the software architecture?
danielvaughn 3y ago
I feel like the core problem here is one that can never be resolved, because it's the result of two valid but opposing forces. On one hand you have business stakeholders, who can see the business reality - money doesn't grow on trees and runways are very real and scary, so without deadlines companies fundamentally cannot function. On the other hand you have engineers, who either explicitly or implicitly understand the other truth - that reliably estimating a project requires knowing all the steps involved, and if you know all the steps involved, then you'd have already automated it.
- dustingetz 3y ago
  as you essentially say, software isn’t solved yet, so the business wins this one for now
- klodolph 3y ago
  When the problem can never be resolved, you redefine the problem until you find a version of the problem with a good solution. So if the problem is that software is hard to estimate, which is hard, we redefine the problem to something like “I want to be able to ship something people want to buy, before we run out of money” or “I want to ship a game in time for Christmas” or “I want to reduce operating costs so we don’t drown in debt.” Incidentally, the process of writing a software program is often the same process—redefining the problem until you find a problem with a good software solution, rather than finding a hard problem and throwing software at it until you die from exhaustion.
  - danielvaughn 3y ago
    I love this.
  - iopq 3y ago
    > “I want to be able to ship something people want to buy, before we run out of money” Buddy, if I knew the answer to the question "something people want to buy" I would be rich already
    - 3y ago
dboreham 3y ago
No something Sherlock.
denvaar 3y ago
I like Basecamp’s “Shape Up” guide https://basecamp.com/shapeup https://basecamp.com/shapeup
juve1996 3y ago
Yea, effort estimation is a waste of time much like building a piece by piece project plan of anything. The key is workflows, like how buildings used to be built. You just keep working on it - no stupid schedule that fails immediately once one little detail to follow goes wrong. Building accurate plans never works and it takes ridiculous amounts of time.
wellpast 3y ago
I might be in a 1% of engineers that believe in estimation. Can you make a step plan for some deliverable? No? Then, sorry, it's not that planning is not possible, it is that you are simply not experienced enough. That's fine. There is something called skills, and you can go get them. Have you made a plan? Great, then is there is some task in your plan that you can't estimate? In rare cases that might be inherently true, but in the majority of cases if you are senior enough any given task you're asked to do at Company A is going to be something similar to some task you've done in your prior experience and you can collect personal data on that. A plan with per task estimates and you have an estimate for a deliverable. The next piece, and this is crucial, is you aim at and execute rigorously to the plan -- prioritizing as you go. If you do any "padding" for insurance, you DO NOT add arbitrary constants or double etc to your estimates. You add extra scope, ie tasks delivering real value, to your plan, but scope that can be cut, if needed, as part of your rigorous execution, en route, if you find in iterations that you're slipping.
- qprofyeh 3y ago
  I tend to disagree. Estimate this more complex situation: Integrate vendor B’s API where you don’t have prior experience with vendor B’s product let alone their API, and where our system is built specifically around vendor A’s API and workflow. This requires backend and frontend (customer and admin UI) changes given that everything is tied to vendor A’s API currently.
  - IanCal 3y ago
    Have you ever done this before? How long has it taken in the past? What are the key first steps? Investigate the API and product right? How long has that taken when you've done it before? Your first deliverable is a plan of it's that unknown. Now you've investigated you know more about what needs to be done. You can compare it to previous times you've done similar things and estimate how long it should take.
    - qprofyeh 3y ago
      Your basic approach is sound. Problem is the existence of unknown unknowns, those that only appear when you reach a later stage where these unknown unknowns decide to present themselves. Often causing an unknown amount of unknown effort requires from an unknown team member or third party. This invalidates earlier timeframes, and in worst case even the entire strategy. How does one put a proper estimate on this? Any answer is but a wild guess. (This is where PMs provide added value by realigning business and product teams around the new situation.)
keyle 3y ago
It's not fake research. It's actually quite an established science in the 24 years I've been doing it. Take your first guess, double it, double it again if the stakeholder is a poser, add 20% per developer less experience than you, subtract 10% for the features you're going to essentially copy paste, add 15% for sick leave (browsing HN) and then double it for every question you have that are unresolved and divide it by the room temperature multiplied by the amount of people with mechanical keyboards. That gives you roughly the right estimate for any job, until the next sprint.
- drunkenmagician 3y ago
  Wait! this is my method, and its patented! You need a licence sir ;)
- cardy31 3y ago
  In this vein, my manager doesn’t appreciate when they ask for an estimate and I say “If I knew how to accurately estimate software projects I would be making millions as a consultant. But I don’t so I’m here”
  - catchnear4321 3y ago
    likely because of your assumption that high paid consultants make accurate estimates. they do get paid more.
    - cardy31 3y ago
      No my assumption isn’t that consultants make good estimates. It is that nobody in our entire industry knows how to accurately estimate how long a software project will take, so I would be a highly sought-after rare commodity in our industry.
- eftychis 3y ago
  The real problem is not with estimation but with the management denial. I was asked to estimate a project I owned as a “green” lad. I offered 6 months with a real detailed plan and up to a year with some extras. Based on past rates we had etc. I even did some statistical modeling. They disagreed. They decided to alter the approach, that the principal engineer had suggested and “it would take 2-3 months.” Two years later I learned they were still working to finish it. Another example: they wanted a Byzantine Fault Tolerance rewrite along with the networking layer. I said we need 6 months. They said we know how it should be done in 1-2 months. I finished my stuff and moved on — they picked couple of people to work on the rest that had said 1-2 months. It took them a year to reach a point they could test it. Now why did I say 6 above? Because that is a project I have done multiple times, I can do everything in my sleep and the delay beyond all other fundamentals is random interruptions. Did that matter? No. When top head works with wishful thinking estimates are ways to find scapegoats; nothing more.
robbles 3y ago
Interesting that most of the top comments here seem to be about the day to day process of estimation itself, when the linked article is about how poorly the research on estimation has been done to date. Did any of you actually read the article?
- OneMoreCoin 3y ago
  Glad I am not the only one who noticed this, thought I was reading the wrong thread. Software engineering research is something I was interested in a long time ago, so hoped for a better, more on-topic discussion.
nprateem 3y ago
My conclusion is there are 2 kinds of engineer: good ones and shit ones. Good ones just need to be told what to do and be left to get on with it. They value autonomy, and any attempt to micromanage them with agile bullshit drastically demotivates them. Shit ones are shit. They need a good one to help them get through their tickets, by asking what's taking so long and showing them better ways of approaching the problem. They need more handholding and don't yet find ticket-sizing a soul-sucking waste of time. Shit ones might become good ones over time, or they might not. The problem is that scrum masters are either former shit engineers or non- technical, so cling dearly to the belief they add some value to the process by acting like a parent (which only benefits the shit ones) or sizing things which benefits no one as it takes too long and is too inaccurate to be useful. Worse, it just demotivates good engineers. Instead, teams need a mix of good and shit engineers, quick standups whose purpose is for the good engineers to realise shit ones need help, and very high level rough estimates at the project level of "this'll probably take us about 3 months". Then get out of their way but be there if they have a blocker. Good engineers will tell you they need help, they don't need a frigging question every morning as if they're 5.
- mcmcmc 3y ago
  As someone who pulled part-time scrum master duties in a seed stage startup, the problem is that engineers, even good ones, tend to be not-so-great communicators. Having someone to grease the wheels in stand ups and planning meetings decreases the time it takes for issues to surface. The other half of the job is cleaning the backlog and keeping the product owner from derailing development. It’s more about waste reduction than adding value. Granted this was just my very limited experience, and we didn’t spend much time on estimating and standups consisted of a scheduled daily Slack ping instead of a meeting since we had major timezone offsets. People over process and all that.
  - regularfry 3y ago
    Not only that. Agile processes in general are supposed to protect everyone from sinking a tonne of effort into successfully building the wrong thing. It doesn't matter if people aren't going as fast as they otherwise might, if the counterfactual has them going in completely the wrong direction. Scrum itself is more specifically designed to protect the team from management interference. Engineers might think it's a waste of time preventing them from getting things done, but in its absence (where it's truly needed) what would actually be happening is constant micromanagement, context switching, and death-march deadline pressure. In its originally-intended form it's not an arbitrary collection of hand-holding niceties, it's a self-defence measure.
throwawaaarrgh 3y ago
Why do software people think they're special? Like it's somehow expected that we can predict everything because we're computer geniuses? We would have accurate estimates if only we "scienced" better? If you want to get very good at estimation, do it for a very long time. Then you will see how your estimates would be very accurate if all the other things in the world didn't happen. Just let go of the idea that you will ever have control over this aspect of life.
- eru 3y ago
  Some industries are pretty good at estimating effort. Eg when a film crew shoots, they have so many weeks, and they usually manage to shoot the footage for the movie within that time.
  - precommunicator 3y ago
    Sure. But software is infinitely more complex than movie or any pure hardware engineering
    - eru 3y ago
      I agree. But the original comment I replied to said: > Why do software people think they're special? And the answer to that is: software is complicated.
  - borplk 3y ago
    One of the big reasons that software estimation doesn't work well is because due to the virtual nature of software it is way too easy to move the goal post in subtle or not subtle ways. In "real world" projects such as civil engineering everything is a lot more stable. You don't start with the intention of building a bridge and decide to turn it into an airport halfway through the project. In software projects one way or another people do frequently keep changing things so the parameters are always changing from under your feet.
    - sh34r 3y ago
BillyTheKing 3y ago
Estimates are always portrayed as 'business reality' vs. 'software engineering preferences' - in my experience using estimates many times actually impedes businesses from measuring other metrics that are just as valuable or (imo) significantly more valuable than estimates. Those are a) 'full' lead-times for features, the coding bit is only a small bit in the whole process and estimating just this bit doesn't really tell the business much about how much effort, time and money it really takes to release a feature and b) value-created for the user by a new feature. Those are harder to do, but usually worth it
ghotli 3y ago
At least when I was younger I was so perplexed by the kind of co-workers that have apparently chosen the profession of working with estimates from other people but continually work without a "confidence level" associated with each estimate. ("two weeks") means such a different thing in reality from ("two weeks", "confidence of 50%"). It's a tuple, it's always been a tuple, it's just downright silly to accept an estimate without the second half of the information. Like what diviniation is going on in that spreadsheet to communicate to others when it doesn't include half the meaningful information to begin with.
- robjan 3y ago
  In this case why not just divide the estimate by the confidence to get the "true estimate" e.g in your example it's actually a 4 week. If you provide a tuple the confidence will eventually be conveniently dropped in some slide deck then the "2 weeks" will be socialised as the commitment.
  - pph 3y ago
    The estimate-confidence relation is not linear. In my experience it follows some kind of S-curve, with 0% confidence approaching impossibly short times and 100% approaching infinity (like a logistic function).
    - staunton 3y ago
      The logistic function is bounded, so I'm not sure what you mean...
      - trashtester 3y ago
        Time spent is on the x-axis and confidence on the y axis. To reach 100% confidence, you have to supply an infinite estiamate.
    - trashtester 3y ago
      I think software projects are best estimated as gamma distributions. To express that as confidence levels, you use the cumulative distribution function of the gamma, which are kind of S shaped, but stretched to the right. https://en.wikipedia.org/wiki/Gamma_distribution https://en.wikipedia.org/wiki/Gamma_distribution
galaxyLogic 3y ago
To estimate how much time something takes on average you need to repeat and measure it many times. For instance if you had a clear unambiguous spec of what the the software must do, you could give it to 10 different teams and measure how long each took. If you then modify the spec even slightly you could make it impossible to create such a program no matter how long you work on it. But every project has its own different spec. We don't know how long it will take until we try to do it at least once. Researches might study how long it takes on average to implement a given specific spec by different teams. But they can not estimate how much it would take to implement any arbitrary spec because - nobody has every tried to implement that spec before. And studying that tells you very little about how much time and effort is needed to implement any other arbitrary spec.
themerone 3y ago
I spent 6 months on a feature that I thought would require 2 lines of code and less than an hour to test and ship. It took 1 month dissecting spaghetti to conclude the task was impossible without a hughe rewrite. 5 months later, the feature was a 2 line change, that could be tested and shipped in an hour.
- deely3 3y ago
  Thats true. For system complex enough you will spend more time learning system than modifying it. Thats why all developers love to write new code and mostly hate to suport existing, working, legacy code.
- e12e 3y ago
  This is the way: > for each desired change, make the change easy (warning: this may be hard), then make the easy change - Kent Beck https://twitter.com/KentBeck/status/250733358307500032 https://twitter.com/KentBeck/status/250733358307500032 Also related to tfa: > If you have a month to plan a project in detail, spend it on four one-week iterations developing while you improve your estimates. If you have a week to plan a project, hold five one-day iterations. Feedback cycles give you information and the experience to make accurate estimates. - Kent Beck, Extreme Programming Explained: Embrace Change (The XP Series)
pcblues 3y ago
I got better at software project estimation the longer I spent in the dev industry. But I still knew I was using back-of-brain knowledge and probably wouldn't benefit from bringing too much of it into my consciousness. I think it's like kicking a goal from a free kick in soccer. Conscious deliberation seems to be a distraction at best.
im_not_robot 3y ago
But how long does ChatGPT estimate?
jillesvangurp 3y ago
A bad estimate is better than no estimates at all. There are plenty of empirical studies that prove that. Some computer scientists have a hard time dealing with non mathematical proofs. They have a huge blind spot for things like empirical case studies, qualitative studies, etc. Those are tools that people in other fields use when mathematical models fall short. There are not a lot of useful mathematical models you can use to model software development. So, understanding that software development is an inherently social process kind of points you in the direction of where to look for alternatives to that. Humans are bad at estimating stuff. Software engineers doubly so. But even a bad estimate gives you some handle on how easy or hard something is to do. And you can break things down to improve the quality of the estimates. And then the rest is just using statistics to model the margins of error. Economists are very good at working with this sort of stuff. I always recommend people look at Don Reinertsen's work on lean development. He's using a lot of economical reasoning and notions such as cost of delay, amount of work in progress, and indeed cost estimates. Those being bad is less of an issue if you know they are only off by 2x vs. 10x. A 2x margin of error gives you some handle on the situation. A 10x uncertainty margin means you need to use some tricks to improve the quality. There are ways to do that. 2x is not great but is better than ignoring estimates.
- doctor_eval 3y ago
  > But even a bad estimate gives you some handle on how easy or hard something is to do Can’t agree with this at all. A bad estimate is by definition bad. How does it give you a handle on anything if it’s unreliable? It’s like saying bad medicine is better than no medicine. And when you talk about breaking an estimating problem into smaller estimates, you’re now firmly in the domain of actually doing the work. An economist can’t take a software problem and decompose it into tractable solutions. Even if they could - good luck getting the customer to pay for that. That’s why I say estimating is an economic problem. The cost of accurate estimates asymptotically approaches the cost of the implementation. Nobody is willing to pay for accurate estimates, and anyway doing this doesn’t make sense. Unfortunately there are very few levers to pull in terms of software planning. Time based project management is way over emphasised. This is why approaches like lean (MVP, CD) and agile exist in the first place.
dang 3y ago
Discussed at the time: Software effort estimation is mostly fake research - https://news.ycombinator.com/item?id=25825244 https://news.ycombinator.com/item?id=25825244 - Jan 2021 (308 comments)
amelius 3y ago
Can't we let an LLM solve this problem? A nice feature for Copilot?
zby 3y ago
What mostly worked for me was two simple things: splitting the big tasks and constantly evaluating evaluations. With more smaller evaluations negative and positive errors anihilate each other. With adjusting the evaluations you can reach the a point where you don't have errors on only one side. A very good book on software estimation is: https://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351 https://www.amazon.com/Software-Estimation-Demystifying-Deve...
mjburgess 3y ago
The best estimate for how long it will take to build X, is to build X many times -- under randomised control conditions -- and take an average. Anything short of this isn't amenable to the scientific method.
- offices 3y ago
  Lots of life depends on information discovered without the scientific method.
  - mjburgess 3y ago
    I agree.
peter-mueller 3y ago
I've started to use the Log-Normal distribution for estimation. Because I think of development like planning a travel time, the train will either arrive in time or the will be a random amount of interruptions/errors that add up. But there won't be as much stuff that speeds the train up (wins back some time). Then I try to communicate the distribution in words or show it, which most often has a hard time to be accepted, because managers expect a normal distribution mean. I let several people estimate large scale features and keep their variance to feed it into the log normal. Often I use a reference project, let the use smaller numbers (but not Fibonacci numbers [1]) to estimate and calculate it back to work days. [1] I personally think the Planning Poker numbers are a lay man's way to force a Log-Normal distribution in the estimation
- amai 3y ago
  See also https://stackoverflow.com/questions/9362286/why-is-the-fibonacci-series-used-in-agile-planning-poker https://stackoverflow.com/questions/9362286/why-is-the-fibon...
rswail 3y ago
There are two accurate forms of software development effort estimation: 1. Guess, double it, and raise it to the next time unit. Example: a 1 day task will take 2 weeks 2. The management inspired "What's the earliest date you can't prove you won't be finished by" system of end date scheduling.
rqtwteye 3y ago
Think about it for five minutes. Whatever number you come up with, multiply by 5. Estimate done. The only problem is that management won’t want to listen.
diamondfist25 3y ago
How did Elon and his companies dealt with this?
AnimalMuppet 3y ago
I remember when I was a kid, being on a family vacation, driving up I-15 in Idaho. At one particular place we topped over a hill, and the road was straight from there to an overpass across the road way down in the distance. I wondered how my dad could aim so well that he'd go under that overpass all the way down there. Now I know that he didn't aim that carefully. Instead, he kept steering. Estimates are estimates. Sure, try to get them approximately right. (By the way, you can't get better if you don't close the loop. Look at estimates vs. reality on your own estimates in the past.) But even good estimates will only be somewhat close to right. You need management that will steer all along the road from estimate to reality. You also have to give them the feedback information so that they can steer. Your estimates shouldn't be off by a factor of 10. "Off by a factor of two" isn't great, but it's not horrible - though they shouldn't always be off by a factor of two in the same direction. If management understands that that's what they're getting, and they know how to work with that, fine. If they think they're getting much more precision than that, that's a management problem. (This is an oversimplification. For some kinds of work, you or your team or company has done that kind of work many times before, and you can easily be more accurate than a factor of two. Other times you need a research project just to identify the parts involved, then another research project to discover whether you can solve those parts, then yet another research project to be able to estimate how long those parts will take. Then you can give an estimate, and it still won't be a very precise one.)
NovemberWhiskey 3y ago
I can give you some kind of reasonable estimate for precisely one kind of work: stuff for which I have good domain knowledge, unfettered access to a small set of stakeholders who are in unambiguous agreement on the need, that is delivered on a technology platform that I know well, using people with whom I've worked in the past, and that has no dependencies on other teams. This has happened approximately three times in my career.
kimjune01 3y ago
the point of estimation is the estimating activity that allows you to prioritize the task over others, not to end up with a useful document
amai 3y ago
I once developed an „AI“ trying to predict when JIRA tickets in our startup get resolved. It was mainly a system based on k-nearest neighbor regression, predicting the closing date based on the average of k similar tickets (we had more than 1000 closed tickets already). It worked reasonable well. One of the most important features was the person creating the ticket. Some people wrote tickets really well and the tickets often got resolved in 48h. Others just wrote one cryptic sentence and nothing happened with these tickets for weeks. But you need a lot of high quality data for that. Since that time I realized that writing wellformed tickets and also changing their state on time is really important for such tools. Modern tools actually don‘t use KNN, but some of them use Monte-Carlo-Simulation to estimate work: https://kanbanize.com/kanban-resources/kanban-analytics/monte-carlo-simulation https://kanbanize.com/kanban-resources/kanban-analytics/mont... https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch... I believe used correctly these tools work better than all those scrum-pokers and t-shirt size manual estimation.

326 points by greghn 3y ago

yacine_ 3y ago

so true

aetherson 3y ago

Site seems hugged to death.

Espressosaurus 3y ago

Site is unresponsive. Internet Archive link: https://web.archive.org/web/20230615230250/https://shape-of-code.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://web.archive.org/web/20230615230250/https://shape-of-...

29athrowaway 3y ago

Even this has more merit than COCOMO https://fliptomato.wordpress.com/2007/03/19/medical-researcher-discovers-integration-gets-75-citations/ https://fliptomato.wordpress.com/2007/03/19/medical-research...

satisfice 3y ago

The datasets are not even data in the first place. If you want to understand estimations and projects, you have to study them. This is a social science project. It's not a math game. It's not a problem that you can solve by collecting oversimplified numbers from dubious sources based on incoherent definitions of the units of measurement. If somebody reports that they spent x days developing a component, what does that mean? What were they doing during those days? How do we know what they were doing? We don't know unless a researcher was right there in the room taking notes in real-time. There are so many confounding variables involved, even if your data collection is perfect, that analysis of the data is a process fraught with opportunities for accidental overfitting or researcher bias.

readthenotes1 3y ago
The one bit of project estimation research that I actually believed was that project estimates are closely affected by timeline desires of major finders/"stakeholders". Anecdata: I once asked a VP of a major telecom why the project deadline was Sept 30 and not Oct 1 or Sept 28, and he replied "because my quarterly bonus depends on it"

awb 3y ago

Ran a successful digital agency for years. Estimation was mostly guesswork. There’s a theory that work expands to fill the time you give it and I found that to be largely true. Other agencies would quote 3 months for projects we’d complete in 6 weeks. Even then I was taking engineers 3 week estimates and doubling them for buffer. We didn’t do anything special other than commit individuals to a single project, or two at most. We would of course add additional resources if necessary to hit deadlines and then adjust future estimates as needed. After a few years estimation felt more instinctual than scientific.

michelpp 3y ago
> There’s a theory that work expands to fill the time you give it https://en.m.wikipedia.org/wiki/Parkinson's_law https://en.m.wikipedia.org/wiki/Parkinson's_law
- Solvency 3y ago
  It can also just barely squeeze into the time you give it, by the skin of your teeth.
  - someweirdperson 3y ago
    The trick is to avoid that people who are doing the work are aware of the estimates or deadlines.
evantbyrne 3y ago
I suspect that experience is a major factor as well. You somehow knew to double those estimates. At the agencies I've worked at over the years, less experienced individuals seemed to consistently underestimate the time it would take to do things and then complete those tasks haphazardly/late/bugged. I went freelance and haven't had any trouble providing accurate estimations.
- mgkimsal 3y ago
  I would guess a repeatable/known type of service or project helps a lot as well. Most of my engagements are relatively different from each other, and I'm doing different things all the time. I've worked places which were offering a narrower set of functionality, with limited customizations, and that's generally much easier to estimate. Also much easier to have estimates in which you've said 'no' to things altogether, vs spending time researching/testing/trialing stuff that's new to you.

quickthrower2 3y ago

My instinct is that if you can put numbers and predictions based on real world estimations, then you are estimating simple things. For example, you can estimate how long it will drive from A to B. Google maps does a brilliant job of this, showing the busy sections. Often I get there +/- a minute from what is estimated over a 60 minute journey. But estimating how long someone will take to do a project, where the project by definition is something new in the context of the people doing it. If it wasn't then the code would already exist, and the estimate will be "how long to deploy that thing you already have using CI". So in software you are always doing something new. We are building Sydney Opera Houses, not prefab housing estates.

x86x87 3y ago
Also, to get better estimates you need to sometimes do some [research] work. Without it ykur guesses are even wilder but with it you have to ack that you've spent time on it, evenif you never get to do the actual work.
- ethbr0 3y ago
  I.e. spiral https://en.m.wikipedia.org/wiki/Spiral_model https://en.m.wikipedia.org/wiki/Spiral_model
  - x86x87 3y ago
    TIL. Thanks for sharing this
    - ethbr0 3y ago
      Sure! I tossed some thoughts on the sibling comment above yours.
  - quickthrower2 3y ago
    That is interesting. Going to look more into this. When something looks like common sense codified, then I like it :-)

mechanicker 3y ago

Reminds me of my early days as a rookie people manager. I was asked to estimate a project for a team that I had no idea about their abilities. I was very hesitant to estimate since I had no data to begin with. My manager asked me to throw a number and see if there is a push back!!! I soon made a decision not to be a manager responsible for guess work based estimation and moved back to an IC role where I estimate effort for work assigned to me.

ElFitz 3y ago

Whenever the topic of software estimates comes up, I am reminded of Erik Bernhardsson’s piece[1]. It was a great read, and still proves useful when trying to explain how things can sometimes go quite wrong. [1]: https://erikbern.com/2019/04/15/why-software-projects-take-longer-than-you-think-a-statistical-model.html https://erikbern.com/2019/04/15/why-software-projects-take-l... Edit: it has come up a few times on hn. See https://news.ycombinator.com/item?id=19671673 https://news.ycombinator.com/item?id=19671673

e12e 3y ago
This is great, thank you. Another approach is to do forecasting based on historical throughput (eg tasks/day). Generally using "story points" or other guesstimates turn out to be redundant if you have actual data (actual project by mostly the same team). It takes into account stuff like n% of issues "blowing up" into subtasks, and some issues taking longer. I read a great article on the subject which I've unfortunately misplaced - but the general gist is touched on in these: "Forecasting software project’s completion date through Monte Carlo Simulation" https://towardsdatascience.com/forecasting-software-projects-completion-date-through-monte-carlo-simulation-c1baa5bcf976 https://towardsdatascience.com/forecasting-software-projects... "When will it be done? A data-driven approach to estimating project completion" https://screenful.com/blog/when-will-it-be-done-a-data-driven-approach-to-estimating-project-completion https://screenful.com/blog/when-will-it-be-done-a-data-drive... "When Will It Be Done: Project Forecasting in Kanban(ize)" (product ad) https://kanbanize.com/blog/project-forecasting/ https://kanbanize.com/blog/project-forecasting/ "Project Forecaster" https://github.com/rodrigozr/ProjectForecaster/wiki https://github.com/rodrigozr/ProjectForecaster/wiki Ed: forgot this one linked from forecaster above: ”#NoEstimates Project Planning Using Monte Carlo Simulation" https://www.infoq.com/articles/noestimates-monte-carlo/ https://www.infoq.com/articles/noestimates-monte-carlo/ And maybe this project is of interest: https://microsoft.github.io/forecasting/ https://microsoft.github.io/forecasting/ In general I'd like to be able to just point a tool at N "completed" GitHub projects, and a fresh one - and get a forecast that adjusts as issues are completed and added (possibly filtering on issue tags in completed and new project) - but I'm not aware of anything out of the box for that.
- 3y ago

pyrolistical 3y ago

Lost me at the first sentence > Effort estimation is an important component of any project, software or otherwise. The most “important component”, what does that even mean? Do they mean task? Isn’t the most important thing to deliver results? Which could mean working software or aborting bad projects

am44jnsf 3y ago
it says "an important", not "most important". It's correct, it is an important.

MuffinFlavored 3y ago

If your manager says "I need an estimate" and you can't tell your manager "no, that's fake research/not worth my time", does it really matter what is/isn't "fake research"?

scraptor 3y ago
Does your manager expect you to obtain your estimate using a machine learning approach from a research paper?

neodypsis 3y ago

> Agile techniques are widely used these days, and datasets from the 1990s are not applicable. What datasets do researchers use to build Agile effort estimation models? In an undergraduate software engineering course I took we learned an effort estimation methodology that assumed a particular way of working (similar to Agile), but also a particular software architecture. The problem was that students then had to adopt such an architecture, even though it wasn't the best fit to the problem at hand, just so that they could complete the effort estimate. Are there methods that are independent of the software architecture?

danielvaughn 3y ago

I feel like the core problem here is one that can never be resolved, because it's the result of two valid but opposing forces. On one hand you have business stakeholders, who can see the business reality - money doesn't grow on trees and runways are very real and scary, so without deadlines companies fundamentally cannot function. On the other hand you have engineers, who either explicitly or implicitly understand the other truth - that reliably estimating a project requires knowing all the steps involved, and if you know all the steps involved, then you'd have already automated it.

dustingetz 3y ago
as you essentially say, software isn’t solved yet, so the business wins this one for now
klodolph 3y ago
When the problem can never be resolved, you redefine the problem until you find a version of the problem with a good solution. So if the problem is that software is hard to estimate, which is hard, we redefine the problem to something like “I want to be able to ship something people want to buy, before we run out of money” or “I want to ship a game in time for Christmas” or “I want to reduce operating costs so we don’t drown in debt.” Incidentally, the process of writing a software program is often the same process—redefining the problem until you find a problem with a good software solution, rather than finding a hard problem and throwing software at it until you die from exhaustion.
- danielvaughn 3y ago
  I love this.
- iopq 3y ago
  > “I want to be able to ship something people want to buy, before we run out of money” Buddy, if I knew the answer to the question "something people want to buy" I would be rich already
  - 3y ago

dboreham 3y ago

No something Sherlock.

denvaar 3y ago

I like Basecamp’s “Shape Up” guide https://basecamp.com/shapeup https://basecamp.com/shapeup

juve1996 3y ago

Yea, effort estimation is a waste of time much like building a piece by piece project plan of anything. The key is workflows, like how buildings used to be built. You just keep working on it - no stupid schedule that fails immediately once one little detail to follow goes wrong. Building accurate plans never works and it takes ridiculous amounts of time.

wellpast 3y ago

I might be in a 1% of engineers that believe in estimation. Can you make a step plan for some deliverable? No? Then, sorry, it's not that planning is not possible, it is that you are simply not experienced enough. That's fine. There is something called skills, and you can go get them. Have you made a plan? Great, then is there is some task in your plan that you can't estimate? In rare cases that might be inherently true, but in the majority of cases if you are senior enough any given task you're asked to do at Company A is going to be something similar to some task you've done in your prior experience and you can collect personal data on that. A plan with per task estimates and you have an estimate for a deliverable. The next piece, and this is crucial, is you aim at and execute rigorously to the plan -- prioritizing as you go. If you do any "padding" for insurance, you DO NOT add arbitrary constants or double etc to your estimates. You add extra scope, ie tasks delivering real value, to your plan, but scope that can be cut, if needed, as part of your rigorous execution, en route, if you find in iterations that you're slipping.

qprofyeh 3y ago
I tend to disagree. Estimate this more complex situation: Integrate vendor B’s API where you don’t have prior experience with vendor B’s product let alone their API, and where our system is built specifically around vendor A’s API and workflow. This requires backend and frontend (customer and admin UI) changes given that everything is tied to vendor A’s API currently.
- IanCal 3y ago
  Have you ever done this before? How long has it taken in the past? What are the key first steps? Investigate the API and product right? How long has that taken when you've done it before? Your first deliverable is a plan of it's that unknown. Now you've investigated you know more about what needs to be done. You can compare it to previous times you've done similar things and estimate how long it should take.
  - qprofyeh 3y ago
    Your basic approach is sound. Problem is the existence of unknown unknowns, those that only appear when you reach a later stage where these unknown unknowns decide to present themselves. Often causing an unknown amount of unknown effort requires from an unknown team member or third party. This invalidates earlier timeframes, and in worst case even the entire strategy. How does one put a proper estimate on this? Any answer is but a wild guess. (This is where PMs provide added value by realigning business and product teams around the new situation.)

keyle 3y ago

It's not fake research. It's actually quite an established science in the 24 years I've been doing it. Take your first guess, double it, double it again if the stakeholder is a poser, add 20% per developer less experience than you, subtract 10% for the features you're going to essentially copy paste, add 15% for sick leave (browsing HN) and then double it for every question you have that are unresolved and divide it by the room temperature multiplied by the amount of people with mechanical keyboards. That gives you roughly the right estimate for any job, until the next sprint.

drunkenmagician 3y ago
Wait! this is my method, and its patented! You need a licence sir ;)
cardy31 3y ago
In this vein, my manager doesn’t appreciate when they ask for an estimate and I say “If I knew how to accurately estimate software projects I would be making millions as a consultant. But I don’t so I’m here”
- catchnear4321 3y ago
  likely because of your assumption that high paid consultants make accurate estimates. they do get paid more.
  - cardy31 3y ago
    No my assumption isn’t that consultants make good estimates. It is that nobody in our entire industry knows how to accurately estimate how long a software project will take, so I would be a highly sought-after rare commodity in our industry.
eftychis 3y ago
The real problem is not with estimation but with the management denial. I was asked to estimate a project I owned as a “green” lad. I offered 6 months with a real detailed plan and up to a year with some extras. Based on past rates we had etc. I even did some statistical modeling. They disagreed. They decided to alter the approach, that the principal engineer had suggested and “it would take 2-3 months.” Two years later I learned they were still working to finish it. Another example: they wanted a Byzantine Fault Tolerance rewrite along with the networking layer. I said we need 6 months. They said we know how it should be done in 1-2 months. I finished my stuff and moved on — they picked couple of people to work on the rest that had said 1-2 months. It took them a year to reach a point they could test it. Now why did I say 6 above? Because that is a project I have done multiple times, I can do everything in my sleep and the delay beyond all other fundamentals is random interruptions. Did that matter? No. When top head works with wishful thinking estimates are ways to find scapegoats; nothing more.

robbles 3y ago

Interesting that most of the top comments here seem to be about the day to day process of estimation itself, when the linked article is about how poorly the research on estimation has been done to date. Did any of you actually read the article?

OneMoreCoin 3y ago
Glad I am not the only one who noticed this, thought I was reading the wrong thread. Software engineering research is something I was interested in a long time ago, so hoped for a better, more on-topic discussion.

nprateem 3y ago

My conclusion is there are 2 kinds of engineer: good ones and shit ones. Good ones just need to be told what to do and be left to get on with it. They value autonomy, and any attempt to micromanage them with agile bullshit drastically demotivates them. Shit ones are shit. They need a good one to help them get through their tickets, by asking what's taking so long and showing them better ways of approaching the problem. They need more handholding and don't yet find ticket-sizing a soul-sucking waste of time. Shit ones might become good ones over time, or they might not. The problem is that scrum masters are either former shit engineers or non- technical, so cling dearly to the belief they add some value to the process by acting like a parent (which only benefits the shit ones) or sizing things which benefits no one as it takes too long and is too inaccurate to be useful. Worse, it just demotivates good engineers. Instead, teams need a mix of good and shit engineers, quick standups whose purpose is for the good engineers to realise shit ones need help, and very high level rough estimates at the project level of "this'll probably take us about 3 months". Then get out of their way but be there if they have a blocker. Good engineers will tell you they need help, they don't need a frigging question every morning as if they're 5.

mcmcmc 3y ago
As someone who pulled part-time scrum master duties in a seed stage startup, the problem is that engineers, even good ones, tend to be not-so-great communicators. Having someone to grease the wheels in stand ups and planning meetings decreases the time it takes for issues to surface. The other half of the job is cleaning the backlog and keeping the product owner from derailing development. It’s more about waste reduction than adding value. Granted this was just my very limited experience, and we didn’t spend much time on estimating and standups consisted of a scheduled daily Slack ping instead of a meeting since we had major timezone offsets. People over process and all that.
- regularfry 3y ago
  Not only that. Agile processes in general are supposed to protect everyone from sinking a tonne of effort into successfully building the wrong thing. It doesn't matter if people aren't going as fast as they otherwise might, if the counterfactual has them going in completely the wrong direction. Scrum itself is more specifically designed to protect the team from management interference. Engineers might think it's a waste of time preventing them from getting things done, but in its absence (where it's truly needed) what would actually be happening is constant micromanagement, context switching, and death-march deadline pressure. In its originally-intended form it's not an arbitrary collection of hand-holding niceties, it's a self-defence measure.

throwawaaarrgh 3y ago

Why do software people think they're special? Like it's somehow expected that we can predict everything because we're computer geniuses? We would have accurate estimates if only we "scienced" better? If you want to get very good at estimation, do it for a very long time. Then you will see how your estimates would be very accurate if all the other things in the world didn't happen. Just let go of the idea that you will ever have control over this aspect of life.

eru 3y ago
Some industries are pretty good at estimating effort. Eg when a film crew shoots, they have so many weeks, and they usually manage to shoot the footage for the movie within that time.
- precommunicator 3y ago
  Sure. But software is infinitely more complex than movie or any pure hardware engineering
  - eru 3y ago
    I agree. But the original comment I replied to said: > Why do software people think they're special? And the answer to that is: software is complicated.
- borplk 3y ago
  One of the big reasons that software estimation doesn't work well is because due to the virtual nature of software it is way too easy to move the goal post in subtle or not subtle ways. In "real world" projects such as civil engineering everything is a lot more stable. You don't start with the intention of building a bridge and decide to turn it into an airport halfway through the project. In software projects one way or another people do frequently keep changing things so the parameters are always changing from under your feet.
  - sh34r 3y ago

BillyTheKing 3y ago

Estimates are always portrayed as 'business reality' vs. 'software engineering preferences' - in my experience using estimates many times actually impedes businesses from measuring other metrics that are just as valuable or (imo) significantly more valuable than estimates. Those are a) 'full' lead-times for features, the coding bit is only a small bit in the whole process and estimating just this bit doesn't really tell the business much about how much effort, time and money it really takes to release a feature and b) value-created for the user by a new feature. Those are harder to do, but usually worth it

ghotli 3y ago

At least when I was younger I was so perplexed by the kind of co-workers that have apparently chosen the profession of working with estimates from other people but continually work without a "confidence level" associated with each estimate. ("two weeks") means such a different thing in reality from ("two weeks", "confidence of 50%"). It's a tuple, it's always been a tuple, it's just downright silly to accept an estimate without the second half of the information. Like what diviniation is going on in that spreadsheet to communicate to others when it doesn't include half the meaningful information to begin with.

robjan 3y ago
In this case why not just divide the estimate by the confidence to get the "true estimate" e.g in your example it's actually a 4 week. If you provide a tuple the confidence will eventually be conveniently dropped in some slide deck then the "2 weeks" will be socialised as the commitment.
- pph 3y ago
  The estimate-confidence relation is not linear. In my experience it follows some kind of S-curve, with 0% confidence approaching impossibly short times and 100% approaching infinity (like a logistic function).
  - staunton 3y ago
    The logistic function is bounded, so I'm not sure what you mean...
    - trashtester 3y ago
      Time spent is on the x-axis and confidence on the y axis. To reach 100% confidence, you have to supply an infinite estiamate.
  - trashtester 3y ago
    I think software projects are best estimated as gamma distributions. To express that as confidence levels, you use the cumulative distribution function of the gamma, which are kind of S shaped, but stretched to the right. https://en.wikipedia.org/wiki/Gamma_distribution https://en.wikipedia.org/wiki/Gamma_distribution

galaxyLogic 3y ago

To estimate how much time something takes on average you need to repeat and measure it many times. For instance if you had a clear unambiguous spec of what the the software must do, you could give it to 10 different teams and measure how long each took. If you then modify the spec even slightly you could make it impossible to create such a program no matter how long you work on it. But every project has its own different spec. We don't know how long it will take until we try to do it at least once. Researches might study how long it takes on average to implement a given specific spec by different teams. But they can not estimate how much it would take to implement any arbitrary spec because - nobody has every tried to implement that spec before. And studying that tells you very little about how much time and effort is needed to implement any other arbitrary spec.

themerone 3y ago

I spent 6 months on a feature that I thought would require 2 lines of code and less than an hour to test and ship. It took 1 month dissecting spaghetti to conclude the task was impossible without a hughe rewrite. 5 months later, the feature was a 2 line change, that could be tested and shipped in an hour.

deely3 3y ago
Thats true. For system complex enough you will spend more time learning system than modifying it. Thats why all developers love to write new code and mostly hate to suport existing, working, legacy code.
e12e 3y ago
This is the way: > for each desired change, make the change easy (warning: this may be hard), then make the easy change - Kent Beck https://twitter.com/KentBeck/status/250733358307500032 https://twitter.com/KentBeck/status/250733358307500032 Also related to tfa: > If you have a month to plan a project in detail, spend it on four one-week iterations developing while you improve your estimates. If you have a week to plan a project, hold five one-day iterations. Feedback cycles give you information and the experience to make accurate estimates. - Kent Beck, Extreme Programming Explained: Embrace Change (The XP Series)

pcblues 3y ago

I got better at software project estimation the longer I spent in the dev industry. But I still knew I was using back-of-brain knowledge and probably wouldn't benefit from bringing too much of it into my consciousness. I think it's like kicking a goal from a free kick in soccer. Conscious deliberation seems to be a distraction at best.

im_not_robot 3y ago

But how long does ChatGPT estimate?

jillesvangurp 3y ago

A bad estimate is better than no estimates at all. There are plenty of empirical studies that prove that. Some computer scientists have a hard time dealing with non mathematical proofs. They have a huge blind spot for things like empirical case studies, qualitative studies, etc. Those are tools that people in other fields use when mathematical models fall short. There are not a lot of useful mathematical models you can use to model software development. So, understanding that software development is an inherently social process kind of points you in the direction of where to look for alternatives to that. Humans are bad at estimating stuff. Software engineers doubly so. But even a bad estimate gives you some handle on how easy or hard something is to do. And you can break things down to improve the quality of the estimates. And then the rest is just using statistics to model the margins of error. Economists are very good at working with this sort of stuff. I always recommend people look at Don Reinertsen's work on lean development. He's using a lot of economical reasoning and notions such as cost of delay, amount of work in progress, and indeed cost estimates. Those being bad is less of an issue if you know they are only off by 2x vs. 10x. A 2x margin of error gives you some handle on the situation. A 10x uncertainty margin means you need to use some tricks to improve the quality. There are ways to do that. 2x is not great but is better than ignoring estimates.

doctor_eval 3y ago
> But even a bad estimate gives you some handle on how easy or hard something is to do Can’t agree with this at all. A bad estimate is by definition bad. How does it give you a handle on anything if it’s unreliable? It’s like saying bad medicine is better than no medicine. And when you talk about breaking an estimating problem into smaller estimates, you’re now firmly in the domain of actually doing the work. An economist can’t take a software problem and decompose it into tractable solutions. Even if they could - good luck getting the customer to pay for that. That’s why I say estimating is an economic problem. The cost of accurate estimates asymptotically approaches the cost of the implementation. Nobody is willing to pay for accurate estimates, and anyway doing this doesn’t make sense. Unfortunately there are very few levers to pull in terms of software planning. Time based project management is way over emphasised. This is why approaches like lean (MVP, CD) and agile exist in the first place.

dang 3y ago

Discussed at the time: Software effort estimation is mostly fake research - https://news.ycombinator.com/item?id=25825244 https://news.ycombinator.com/item?id=25825244 - Jan 2021 (308 comments)

amelius 3y ago

Can't we let an LLM solve this problem? A nice feature for Copilot?

zby 3y ago

What mostly worked for me was two simple things: splitting the big tasks and constantly evaluating evaluations. With more smaller evaluations negative and positive errors anihilate each other. With adjusting the evaluations you can reach the a point where you don't have errors on only one side. A very good book on software estimation is: https://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351 https://www.amazon.com/Software-Estimation-Demystifying-Deve...

mjburgess 3y ago

The best estimate for how long it will take to build X, is to build X many times -- under randomised control conditions -- and take an average. Anything short of this isn't amenable to the scientific method.

offices 3y ago
Lots of life depends on information discovered without the scientific method.
- mjburgess 3y ago
  I agree.

peter-mueller 3y ago

I've started to use the Log-Normal distribution for estimation. Because I think of development like planning a travel time, the train will either arrive in time or the will be a random amount of interruptions/errors that add up. But there won't be as much stuff that speeds the train up (wins back some time). Then I try to communicate the distribution in words or show it, which most often has a hard time to be accepted, because managers expect a normal distribution mean. I let several people estimate large scale features and keep their variance to feed it into the log normal. Often I use a reference project, let the use smaller numbers (but not Fibonacci numbers [1]) to estimate and calculate it back to work days. [1] I personally think the Planning Poker numbers are a lay man's way to force a Log-Normal distribution in the estimation

amai 3y ago
See also https://stackoverflow.com/questions/9362286/why-is-the-fibonacci-series-used-in-agile-planning-poker https://stackoverflow.com/questions/9362286/why-is-the-fibon...

rswail 3y ago

There are two accurate forms of software development effort estimation: 1. Guess, double it, and raise it to the next time unit. Example: a 1 day task will take 2 weeks 2. The management inspired "What's the earliest date you can't prove you won't be finished by" system of end date scheduling.

rqtwteye 3y ago

Think about it for five minutes. Whatever number you come up with, multiply by 5. Estimate done. The only problem is that management won’t want to listen.

diamondfist25 3y ago

How did Elon and his companies dealt with this?

AnimalMuppet 3y ago

I remember when I was a kid, being on a family vacation, driving up I-15 in Idaho. At one particular place we topped over a hill, and the road was straight from there to an overpass across the road way down in the distance. I wondered how my dad could aim so well that he'd go under that overpass all the way down there. Now I know that he didn't aim that carefully. Instead, he kept steering. Estimates are estimates. Sure, try to get them approximately right. (By the way, you can't get better if you don't close the loop. Look at estimates vs. reality on your own estimates in the past.) But even good estimates will only be somewhat close to right. You need management that will steer all along the road from estimate to reality. You also have to give them the feedback information so that they can steer. Your estimates shouldn't be off by a factor of 10. "Off by a factor of two" isn't great, but it's not horrible - though they shouldn't always be off by a factor of two in the same direction. If management understands that that's what they're getting, and they know how to work with that, fine. If they think they're getting much more precision than that, that's a management problem. (This is an oversimplification. For some kinds of work, you or your team or company has done that kind of work many times before, and you can easily be more accurate than a factor of two. Other times you need a research project just to identify the parts involved, then another research project to discover whether you can solve those parts, then yet another research project to be able to estimate how long those parts will take. Then you can give an estimate, and it still won't be a very precise one.)

NovemberWhiskey 3y ago

I can give you some kind of reasonable estimate for precisely one kind of work: stuff for which I have good domain knowledge, unfettered access to a small set of stakeholders who are in unambiguous agreement on the need, that is delivered on a technology platform that I know well, using people with whom I've worked in the past, and that has no dependencies on other teams. This has happened approximately three times in my career.

kimjune01 3y ago

the point of estimation is the estimating activity that allows you to prioritize the task over others, not to end up with a useful document

amai 3y ago

I once developed an „AI“ trying to predict when JIRA tickets in our startup get resolved. It was mainly a system based on k-nearest neighbor regression, predicting the closing date based on the average of k similar tickets (we had more than 1000 closed tickets already). It worked reasonable well. One of the most important features was the person creating the ticket. Some people wrote tickets really well and the tickets often got resolved in 48h. Others just wrote one cryptic sentence and nothing happened with these tickets for weeks. But you need a lot of high quality data for that. Since that time I realized that writing wellformed tickets and also changing their state on time is really important for such tools. Modern tools actually don‘t use KNN, but some of them use Monte-Carlo-Simulation to estimate work: https://kanbanize.com/kanban-resources/kanban-analytics/monte-carlo-simulation https://kanbanize.com/kanban-resources/kanban-analytics/mont... https://www.joelonsoftware.com/2007/10/26/evidence-based-scheduling/ https://www.joelonsoftware.com/2007/10/26/evidence-based-sch... I believe used correctly these tools work better than all those scrum-pokers and t-shirt size manual estimation.