21 ms·

Software effort estimation is mostly fake research

780 points by walterclifford 6y ago

kfk 6y ago
At least in a business setting I think the whole concept of a project needs serious reconsideration. We end up more often than not trying to fit developing a digital product into an enormously stupid gantt chart to execute some poorly thought “business requirements”. I prefer to talk products and not projects, I deliver the full thing including “growth” as adoption doesn’t come “if you build it” even within a Company setting. If you are building a product you can also get closer to those with the real problem willing to fund you with real budgets. On top of everything else if you are making users happy they will not chase you on fake estimates but rather work with you to get stuff done.
- snidane 6y ago
  Funny when you look at the actual definition of a project straight from PMP. "a temporary endeavor undertaken to create a unique project service or result." No "projects" in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption. We in software think of projects really as things which we create and which need maintenance in order to live. There is never some "end" to it. Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.
wyldfire 6y ago
How do other disciplines estimate NRE? Do they have the same problems with missed predictions?
- solumos 6y ago
  Well, lawyers take a retainer. Doctors are paid per-diagnostic-visit and per-procedure. Accountants bill hourly. Most professions have this - electricians, plumbers, locksmiths, etc. This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple "feature factories" (see also: Developer Hegemony[0]). [0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-software-developers-should-run-software-development/ https://daedtech.com/developer-hegemony-the-crazy-idea-that-...
- ghaff 6y ago
  Sure. A lawyer may have a pretty good idea of how a particular task or case is likely to play out but there's a lot that isn't under their control. I imagine something like drafting a will is relatively straightforward but I also imagine a criminal case has a huge number of variables.
- alkonaut 6y ago
  You only quote the fixed bits and charge by the hour for the rest. If you need to give a quote for an unknown you multiply the quote by N to cover a worst case.
csours 6y ago
"Estimates" are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it. Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.
- Const-me 6y ago
  I mostly agree, but for research, estimating the estimation is often good enough in practice. For me it often something like this: I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.
- chaz72 6y ago
  Yes! I try to always phrase it as "the part I can see from here will take at least X time".
- Ericson2314 6y ago
  Yes, and if you find yourself getting better at estimation, that's probably because you have failed to build proper abstractions. With proper abstractions, the cost of the same stuff should be minimal, so the part with no good priors predominates.
- deleted 6y ago
  [deleted]
- Netcob 6y ago
  Very true, I think those estimates are actually two ideas/types crammed into one value. 1. The construction part, as you said, can be estimated. 2. I'd just call the other thing "allocated time" instead of "estimated time". Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem. For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.
domano 6y ago
I noticed that my estimates usually end up 3x as high as the next one and i feel pretty confident about them
jakubp 6y ago
If someone can conclusively teach inexperienced programmers good approach to estimates (methodology) + help embed this into sales process of a software house-type company, I know some folks who'd love to have this :) My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.) But at that point nobody cares! People go on with further projects, and no data is ever collected. Nobody learns. When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away. Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done. It's insane.
- 1337shadow 6y ago
  Make a complete list of things to develop in a spreadsheet, "complete" means every single item the customer wants to see in the product, not only every single button but also every single label, that should be definable by reading the project specs or mockups. I think "forgetting things" is the first big mistake leading to under estimations. Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work. I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV Only problem: the customer might not like what he gets, even if it works exactly as planned.
  - csours 6y ago
    > "complete" means every single item the customer wants to see in the product, Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.
tootie 6y ago
Scrum dogma is that estimates are for complexity, not effort or timing. The points you track in JIRA are meant to reflect how much of the current backlog is complete and how much is remaining. That can be extrapolated into timing but can't be done up front.
- dragonwriter 6y ago
  > Scrum dogma is that estimates are for complexity, not effort or timing The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates. The “dogma” you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.
  - cuillevel3 6y ago
    Right, the estimates are not even in the guide: https://www.scrumguides.org/scrum-guide.html#sprint-planning https://www.scrumguides.org/scrum-guide.html#sprint-planning Planning poker and such are useful as they encourage the team to discuss differences and help identifying stories which are too large.
- wccrawford 6y ago
  My understanding is that for a team and project that's been going for a while, scrum points can be roughly turned into time estimates by looking at the history of stories that were rated for that many points and averaging. But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head. And IME, it's a lot less stressful on the programmer to estimate points rather than time.
- jakubp 6y ago
Aldipower 6y ago
I am 20 years in development business now. This simple rule of thumb works for me and the team: (Your honest and concise estimation) * 3 There are just to many unknowns you cannot foresee. Software development is complex.
- kilroy123 6y ago
  I couldn't agree more. This is what I do.
- OneGuy123 6y ago
  This, the famous "prediction* Pi" rule seems to be correct also in my experience. I make the most optimistic prediction and then * PI.
- curiousllama 6y ago
  I've been hearing this rule for years now, and I think the coefficient is increasing. It started at 1.5x, now it's at 3x... Is this just me?
  - Aldipower 6y ago
    Maybe the coefficient is related to your age. You got older, so there is more wisdom around. ;)
  - impy 6y ago
    I'd say this is Parkinson's Law at play
- kthejoker2 6y ago
  I'm such an optimist! I multiply by 2.2.
- matwood 6y ago
  I use a similar methodology. The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).
neartheplain 6y ago
Reminds me of the excellent 2010 talk, “What We Actually Know About Software Development, and Why We Believe It’s True”: https://vimeo.com/9270320 https://vimeo.com/9270320 Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).
jadams3 6y ago
The most useful piece of advice I have gotten for estimates is that they are all junk until someone sits down and tries to do the work. For big jobs, 20% of it, small jobs pushing 50% of the work. In every single case when you've done that much work, I seem to wind up with a reasonable estimate. If we've done the job before and have data on it, also reasonable. Double or triple anything else.
captain_price7 6y ago
I have been involved in Software engineering research a bit, even have a first-author short paper. I was struck by just how much pointless, low-effort papers there are in this domain. People have been researching about bug prediction for over two decades now, and judging by the paper quantity, this isn't a niche area. Yet how many organizations do actually employ those systems in real-world? Can't comment on industry, but I haven't found a single open-source program that does that [1]. Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research. [1] I saw Mozilla attempt it, but not sure if it's currently in use.
- randomsearch 6y ago
  Couldn't agree more. Software engineering is a very practical, fast-moving subject, and that makes most academic work even more irrelevant - software engineering academics often don't have the first clue about what real software engineering is about, because they don't have recent industrial experience.
ChrisMarshallNY 6y ago
I find this amusing. Know how much a Honda Accord costs? About 25 Grand. Know how much a Mercedes S450 costs? About three times as much. They are both great cars, that will be rewarding to own. The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface. The reason is that all those little things that go into high quality take lots of time. I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot. However, if you want that app to be ship quality, and highly usable, you're looking at over a month. The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.
- darkerside 6y ago
  Your base assumption seems to include that quality is valuable for its own sake. I don't totally disagree, but I'm wary of assigning value based on effort rather than output. Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.
  - ChrisMarshallNY 6y ago
    You are exactly correct. That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale. I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger." Our prototypes were incredibly expensive. A $2,000 (retail) body prototype would be insured at half a million bucks. I just find, in my experience, managers expect Mercedes-level quality, for Kia (not even Honda) prices, and (if you are lucky) mature Honda assembly line development speed. That can lead to real Jurassic-scale disasters. If you want that level of quality, you need to plan for it. I'm working on the app that I'm doing, because I was helping a friend of mine evaluate contractors for his dream. The promises they made were absurd. After a couple of these, I just said "Screw it. I'll do it for you." He'll be getting Mercedes quality, but he'll need to wait a bit longer for it. I am pretty good at doing damn good; damn fast. But he also won't be paying a dime for it, so I think he's good with that.
Netcob 6y ago
Integrations for time-tracking software with issue tracking software exists, but I've never actually seen them used in combination before. You'd think that issue trackers would always include a very user-friendly time tracker by default, and advertise integrations with popular time trackers just to get that "actual time spent" data. I've also never looked at a burndown chart with any particular interest after knowing you can close an 8h issue after .5h and a .5h issue after 8h of work and the issue tracker will just keep pretending its charts mean anything. Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.
458aperta 6y ago
I disagree with the premise of this article. Software development is largely a waterfall process even if you use TDD (which was using a machine gun to chop down a tree) and the effort estimation is both predictable and consistent. At least with the model that we use in-house developed over the past 10 years that has resulted in several exits. Don't rely on HN submissions to shape your view of the world. Pedantry, pendants waste your time. Only real world results and data matter. The people who can't do simply teach or publish papers so all you are left are people who grab a leg or an arm and think its an elephant in complete darkness. You simply will never ever find us publishing these information nor do others that have figured it out. Transparency when it comes to winning trust is a must. Disclosures in the name of generosity or some collective good is lust; You might feel good but it ultimately results in increased competition and higher operating cost until your edge disappears. We are not in the business of helping our competitors, we use our unique edge to push them out of the market when we can, we use capital to buy them out when we can't.
alisaus6 6y ago
Hottie hangout pics with nude babes - https://adultlove.life https://adultlove.life
gpmcadam 6y ago
Cache/mirror http://webcache.googleusercontent.com/search?q=cache:http://shape-of-code.coding-guidelines.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/&client=safari&hl=en-gb&prmd=ivn&strip=0&vwsrc=0 http://webcache.googleusercontent.com/search?q=cache:http://...
Londane 6y ago
The site is down for me, so : https://web.archive.org/web/20210118200327/http://shape-of-code.coding-guidelines.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://web.archive.org/web/20210118200327/http://shape-of-c...
distalx 6y ago
Broken Link....
ThomPete 6y ago
In his book “Code Complete”, Steven McConnel speaks about metaphors. He reasons that metaphors are necessary to be a good developer as it helps visualize the act of coding. The metaphor he prefers is the “construction” metaphor. This metaphor he argues best explain the act (some would say art) of programming and gives developers a language to speak in that brings clarity to the development process. When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc) The problem with that metaphor IMO is that it's not actually what software development is. The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building. And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does. I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.
- randomsearch 6y ago
  The reality is that software development is nothing like engineering or construction, it's totally different. You don't build a quick house, let people live in it and start building the walls whilst they live there. Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.
  - ThomPete 6y ago
    My point was more that the software developer have to not just build with existing material and equipment but have to invent and build those on the fly.
SKILNER 6y ago
Not only do we not know how to predict how long a software project will take, we don't even know how to predict what the end product will look like. So who are we kidding? Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team. About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.
- xpe 6y ago
  Excellent point. Can you share your presentation or at least some of your thinking behind it? Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).
jacques_chester 6y ago
I think a better title would have been "most effort estimation literature relies on small datasets". The use of "fake" can be read as insinuating deliberate malpractice.
dboreham 6y ago
Redirect loop on TFA? Update: working now.
Kaze404 6y ago
I had a conversation about estimates during a recent interview. I asked about how the company deals with those, and the interviewer said they don't do estimates because there's never been a time where something productive came out of one, and I think it makes sense. In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.
- snidane 6y ago
  When a company uses software estimation, it suggests strong distrust towards the software people and is looking for justification of those huge costs. Most often it means some shitshow happened or is still going on in there.
yters 6y ago
It is provably impossible to predict development effort.
k__ 6y ago
I think, software development works best without deadlines, but then you have to find ways to finance yourself while people wait.
didibus 6y ago
The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote. The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business. If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team. Now keep in mind that the "business" in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the "business" will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.
- quickthrower2 6y ago
  That is the degenerate scenario, but it’s not always true. Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it. And I’ve also seen wise use of estimation but that is rarer!
  - didibus 6y ago
    > And I’ve also seen wise use of estimation but that is rarer I've got a question for you here, what is a wise use of estimation? What problems are estimates the ideal solution for? In the context of software development? I can think of only two: planning headcount and priorities (aka planning the order in which you do things). When you focus on those problems, you realize there's no need for an accurate estimate. All you need is relative sizing between requests. This seems twice as much work as that but gives us half the benefit. Ok probably we shouldn't do this one first. And even then, often time impact is all you need for priorities. Like who cares what takes you longer, if you just focus on building the next most impactful thing you'll very often succeed. The question of: "When are we going to be done this?" Is almost always irrelevant except for what I mentioned "asking for a quote". Like, who cares when it's done? What else you'd rather we do? I can tell you if your alternative will be any faster or not. And is this not the most impactful thing? If so, let's finish it and stop slowing us down with these irrelevant questions. Now for headcount, it's the same thing. Look how much you did last year with what headcount. Ask yourself if what you're hoping to do this year is of the same relative size, twice as big, half as big, or smaller? And then plan headcount accordingly. If twice as big, double head count. Overtime you'll learn if twice as big requires doubling headcount or 1.5x or 1.2x or 2.3x etc.
mr_tristan 6y ago
I wish as much attention was paid to perform post mortems regularly then to only do estimation. You know, actually look at "hey, this is what we guessed and this is what actually happened". I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?" So clearly, there's value, at least when we've done them. I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".
- mr_tristan 6y ago
  To tie this a bit more into the actual article: It might be even more accurate to just ask people in a post mortem for some feedback, instead of trying to build some data set based on estimates and SLOC. Like an exit poll.
taeric 6y ago
My problem with this is all estimation is hard. Period. Quantitative discussions of something that has been done? Fairly easy, if still not accurate. Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done. Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?
- edelans 6y ago
  I like this analogy. But I was picturing myself asking my sales manager "How long would it take you to learn to play a song on piano?", and I'm pretty sure he would reply "but you touched a piano before, you are supposed to be a professional piano player ! A professional piano player surely knows how long it would take him to learn how to play a song". So I guess he would miss the point totally :/
  - taeric 6y ago
    Ironically missing that some folks will never be able to learn some songs. And, learning to play an existing song doesn't necessarily translate to being able to write one.
- perl4ever 6y ago
  Estimation is hard if you are stuck in the mindset that you have to make a point estimate. It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that. It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know. It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.
  - valenterry 6y ago
    This! If you estimate, estimate a distribution.
  - 6y ago
vincentmarle 6y ago
I've been using the Rumsfeld Matrix to classify software estimations into 4 categories: Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments). Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code). Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity). Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.
djoldman 6y ago
>Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression. >...non-machine learning techniques, e.g., regression. Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.
- disgruntledphd2 6y ago
  They (almost certainly) mean linear regression, which many people appear to regard as statistics and boring, as opposed to single layer neural networks, which are cool. To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).
MattGaiser 6y ago
Estimation seems like a paperwork generation exercise anyway, so why not take it all the way to research?
meesles 6y ago
It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.
- randomsearch 6y ago
  I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.
  - deleted 6y ago
    [deleted]
  - avdlinde 6y ago
    Well, the metric is #papers and #citations to get funding. So yes that's true, ideally, but...
    - JamesBarney 6y ago
      Relevant blog post - It's not the incentives it's you. https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...
      - angry_octet 6y ago
        Could not disagree more. The premise of this piece is that dieting is just willpower, beating addiction is the same, an entirely individual problem. The reality is that the incentives are what society is asking for, with ethics etc acting as constraints only, rather than actively rewarded. In that formulation it is inevitable (and optimal) that some people will skate to the edge, and if the edge is poorly enforced they will increasingly go over. Policies must address the overall actual effects, not just a chimerical ideal. (It is even more ridiculous coming from someone in psychology.)
lugu 6y ago
I bet it is possible to give precise estimates, the problem is the amount of time and effort to produce such estimation compares with the actual execution.
gregors 6y ago
It's worth watching Software Schedules by Joel Spolsky has some of the best insight regarding evidence based scheduling. It's extremely interesting - https://www.youtube.com/watch?v=EUS4ktQJOSY https://www.youtube.com/watch?v=EUS4ktQJOSY
Groxx 6y ago
One important thing I've managed to get a couple managers to track for quarter/half planning purposes: uncertainty. Prior to that, we'd of course hemmed and hawed and verbalized the fuzziness of our estimates... but only the final decided-on not-super-pessimistic number was written down, and decisions were made based on those. That was a major source of our estimating problems. Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are. It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.
samsk 6y ago
My bulletproof formula for doing estimations in corporate environments: InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order Example: 1 day (MD) = 2 weeks 2 weeks = 4 months
kylecordes 6y ago
Sometimes a request for an “estimate” is really a request for a promise, quotation, a guarantee that something will be delivered by X time or cost. It's easy to detect this: Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%? If you get emotional pushback to discussing uncertainty, they are looking for a promise.
- xpe 6y ago
  Yes, many of us are too likely to interpret the word "estimate" at face value. It is a wonderful idea to view this instead as only a starting point -- an information-gathering conversation -- as how to provide your customer or other stake-holders what they need. If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)
yibg 6y ago
To me the difficulty of estimating software projects isn't about estimating the time / effort to build the thing. The biggest uncertainty is trying to estimate what it is we're trying to build in the first place. i.e. the details of the spec. Often times where I've seen estimates really off is when the scope changes, or more details are discovered etc. To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.
deandree_ 6y ago
Exactly, you can't really predict what can't be predicted
BlargMcLarg 6y ago
I seriously don't understand the obsession with estimates in the paradigm it is pushed in. Want to be steady at a reliable pace, work for a few months and figure out the average and deviations. Need a priority on tasks, use a priority queue using whatever method you assign it by, often a combination of deadline, value added, complexity and more. Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models? Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.
- Jweb_Guru 6y ago
  That is absolutely what it is. For example, people are comparing favorably to how sales reps are now forced to record everything they do in something like Salesforce to try to prove they are productive, which by all accounts is pretty awful for the actual people doing the selling and doesn't actually improve sales--it just makes management feel like they're doing something.
snidane 6y ago
Software development which is a repeatable and already defined process is totally possible to predict and estimate. Most tasks of repeatable processes follow normal distribution and is predictable. Deviations from expected mean will be due to predictable factors of the environment such as failed disk or sleepy programmer. You can apply arbitrary six sigma methodology to measure such process with accuracy. The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another. Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts. Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance. Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity. Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.
- k__ 6y ago
  "a repeatable process would be immediately automated away by writing a function, library, framework or any such tools" This is the crucial point here. Source code is a (almost) self-assembling blueprint. The actual product that will be build is the software and software is a configuration of matter, in this case of a computer. The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house. With better robotics, we will probably get there at some point in the future. And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.
encoderer 6y ago
Don't do an estimate, build a prototype. Don't do an estimate, build a prototype. Don't do an estimate... seriously... build a prototype. And then throw it out. And THEN do an estimate. It will probably be pretty accurate. Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.
mcguire 6y ago
Would software effort estimation work better if software developers were held to fixed-price bids?
tobyhinloopen 6y ago
I feel like I can get pretty good estimates on the following conditions: - the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.
- commandlinefan 6y ago
  > the application is thoroughly specced. You might need WEEKS for this More weeks than it will end up taking to build the finished product, in fact.
mxcrossb 6y ago
I need to get a check list for the ever so popular criticisms of academia article. This one surely would hit plenty of boxes: scientists don’t pay attention to industry, researchers only care about grants, etc. It’s a lot of words just to say “they need bigger datasets”, which is also a check box. The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.
rietta 6y ago
I've been in this business long enough to know that point estimates are always wrong. A proper estimate is a range with a confidence interval. When forced to do a fixed bid, you have to raise the price even higher to the upper end of the cone of uncertainty.
- laichzeit0 6y ago
  This agrees with my experience as well. Stop giving point estimates.
sn_master 6y ago
That's how I always felt about all the estimating parts in CS courses. They were all made up BS that would only work on very pre-defined things, like creating a new DB or website based on a template. aka "IT" and not "SE" stuff.
perl4ever 6y ago
I had access to information about historical projects, so I compared the actual amount of time taken to the estimated time at the beginning, for every software project in the history of my organization. I found that on average, things take twice as long as expected. So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it. A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot. But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.
- commandlinefan 6y ago
  Hofstatder's law: It always takes longer than you expect, even after accounting for Hofstatder's law.
jugg1es 6y ago
I think there is a big difference between building something totally new versus a modification to an existing system. If you are building something totally new, I do not think there is much value in estimates with granularity lower than a month.
deleted 6y ago
[deleted]
franzwong 6y ago
In my previous company, we needed to put project code into timesheet for every activity. Of course, requirement gathering is also an activity. However, before you get the budget, you don't have the project code. Also, you can't get the budget before you have the estimate. It means I needed to give an estimate before I know what system I am going to build.
- valenterry 6y ago
  How about getting a budget to do an estimation/PoC/prototype for a project?
deleted 6y ago
[deleted]
morelandjs 6y ago
My issue with effort estimates is that it is taboo to report estimates with uncertainty (think how silly that sounds). I'll often give a low-end / high-end and the person asking for the estimate will tell me they want a single number. If you did this in physics you would get laughed and for good reason. It's bad science, and it's bad forecasting. You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?
newscracker 6y ago
So research into software estimation also seems to follow Hofstadter's Law: “It always takes longer than you expect, even when you take into account Hofstadter's Law.”
ChicagoDave 6y ago
Unfortunately, the business folks insist on making up arbitrary dates and costs. It’s the way of the world.
galaxyLogic 6y ago
I think the problem is that if you have a timeline you can produce the result within time and budget, but at the expense of quality. The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).
grey-area 6y ago
It is possible to deliver most software on time, but you need to be very clear about scope and unknowns and keep the units of work small. The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools). If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation. When new ideas arise, they should usually be kept for future work. When work is added, other work needs to be removed or the estimate revised. If the work runs late, something should be removed from the scope to keep it on time, then done after delivery. If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.
toolslive 6y ago
Hasn't anyone tried to replace time estimates with probabilities? (Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)
- rightbyte 6y ago
  I don't know if I am supposed to roll a 20 on a D20 or a 8 on a D8 to begin with. Time estimation on a macro or micro level is just educated feelings. The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice. I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.
parentheses 6y ago
The first key mistake of software estimation in industry is that we rarely talk about confidence (e.g “I’m 50% confident it’ll be shipped in 5 hours, but 99% confident it’ll ship before end of week”). The fact that this is not part of planning means it’s often not part of the conversation and that we seldom think about estimation as a tuple of value and confidence. The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.
zomars 6y ago
Jonathan Stark saved my soul from this estimates nightmare. You should check him out!

kfk 6y ago

At least in a business setting I think the whole concept of a project needs serious reconsideration. We end up more often than not trying to fit developing a digital product into an enormously stupid gantt chart to execute some poorly thought “business requirements”. I prefer to talk products and not projects, I deliver the full thing including “growth” as adoption doesn’t come “if you build it” even within a Company setting. If you are building a product you can also get closer to those with the real problem willing to fund you with real budgets. On top of everything else if you are making users happy they will not chase you on fake estimates but rather work with you to get stuff done.

snidane 6y ago
Funny when you look at the actual definition of a project straight from PMP. "a temporary endeavor undertaken to create a unique project service or result." No "projects" in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption. We in software think of projects really as things which we create and which need maintenance in order to live. There is never some "end" to it. Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.

wyldfire 6y ago

How do other disciplines estimate NRE? Do they have the same problems with missed predictions?

solumos 6y ago
Well, lawyers take a retainer. Doctors are paid per-diagnostic-visit and per-procedure. Accountants bill hourly. Most professions have this - electricians, plumbers, locksmiths, etc. This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple "feature factories" (see also: Developer Hegemony[0]). [0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-software-developers-should-run-software-development/ https://daedtech.com/developer-hegemony-the-crazy-idea-that-...
ghaff 6y ago
Sure. A lawyer may have a pretty good idea of how a particular task or case is likely to play out but there's a lot that isn't under their control. I imagine something like drafting a will is relatively straightforward but I also imagine a criminal case has a huge number of variables.
alkonaut 6y ago
You only quote the fixed bits and charge by the hour for the rest. If you need to give a quote for an unknown you multiply the quote by N to cover a worst case.

csours 6y ago

"Estimates" are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it. Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.

Const-me 6y ago
I mostly agree, but for research, estimating the estimation is often good enough in practice. For me it often something like this: I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.
chaz72 6y ago
Yes! I try to always phrase it as "the part I can see from here will take at least X time".
Ericson2314 6y ago
Yes, and if you find yourself getting better at estimation, that's probably because you have failed to build proper abstractions. With proper abstractions, the cost of the same stuff should be minimal, so the part with no good priors predominates.
deleted 6y ago
[deleted]
Netcob 6y ago
Very true, I think those estimates are actually two ideas/types crammed into one value. 1. The construction part, as you said, can be estimated. 2. I'd just call the other thing "allocated time" instead of "estimated time". Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem. For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.

domano 6y ago

I noticed that my estimates usually end up 3x as high as the next one and i feel pretty confident about them

jakubp 6y ago

If someone can conclusively teach inexperienced programmers good approach to estimates (methodology) + help embed this into sales process of a software house-type company, I know some folks who'd love to have this :) My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.) But at that point nobody cares! People go on with further projects, and no data is ever collected. Nobody learns. When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away. Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done. It's insane.

1337shadow 6y ago
Make a complete list of things to develop in a spreadsheet, "complete" means every single item the customer wants to see in the product, not only every single button but also every single label, that should be definable by reading the project specs or mockups. I think "forgetting things" is the first big mistake leading to under estimations. Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work. I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV Only problem: the customer might not like what he gets, even if it works exactly as planned.
- csours 6y ago
  > "complete" means every single item the customer wants to see in the product, Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.

tootie 6y ago

Scrum dogma is that estimates are for complexity, not effort or timing. The points you track in JIRA are meant to reflect how much of the current backlog is complete and how much is remaining. That can be extrapolated into timing but can't be done up front.

dragonwriter 6y ago
> Scrum dogma is that estimates are for complexity, not effort or timing The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates. The “dogma” you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.
- cuillevel3 6y ago
  Right, the estimates are not even in the guide: https://www.scrumguides.org/scrum-guide.html#sprint-planning https://www.scrumguides.org/scrum-guide.html#sprint-planning Planning poker and such are useful as they encourage the team to discuss differences and help identifying stories which are too large.
wccrawford 6y ago
My understanding is that for a team and project that's been going for a while, scrum points can be roughly turned into time estimates by looking at the history of stories that were rated for that many points and averaging. But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head. And IME, it's a lot less stressful on the programmer to estimate points rather than time.
jakubp 6y ago

Aldipower 6y ago

I am 20 years in development business now. This simple rule of thumb works for me and the team: (Your honest and concise estimation) * 3 There are just to many unknowns you cannot foresee. Software development is complex.

kilroy123 6y ago
I couldn't agree more. This is what I do.
OneGuy123 6y ago
This, the famous "prediction* Pi" rule seems to be correct also in my experience. I make the most optimistic prediction and then * PI.
curiousllama 6y ago
I've been hearing this rule for years now, and I think the coefficient is increasing. It started at 1.5x, now it's at 3x... Is this just me?
- Aldipower 6y ago
  Maybe the coefficient is related to your age. You got older, so there is more wisdom around. ;)
- impy 6y ago
  I'd say this is Parkinson's Law at play
kthejoker2 6y ago
I'm such an optimist! I multiply by 2.2.
matwood 6y ago
I use a similar methodology. The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).

neartheplain 6y ago

Reminds me of the excellent 2010 talk, “What We Actually Know About Software Development, and Why We Believe It’s True”: https://vimeo.com/9270320 https://vimeo.com/9270320 Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).

jadams3 6y ago

The most useful piece of advice I have gotten for estimates is that they are all junk until someone sits down and tries to do the work. For big jobs, 20% of it, small jobs pushing 50% of the work. In every single case when you've done that much work, I seem to wind up with a reasonable estimate. If we've done the job before and have data on it, also reasonable. Double or triple anything else.

captain_price7 6y ago

I have been involved in Software engineering research a bit, even have a first-author short paper. I was struck by just how much pointless, low-effort papers there are in this domain. People have been researching about bug prediction for over two decades now, and judging by the paper quantity, this isn't a niche area. Yet how many organizations do actually employ those systems in real-world? Can't comment on industry, but I haven't found a single open-source program that does that [1]. Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research. [1] I saw Mozilla attempt it, but not sure if it's currently in use.

randomsearch 6y ago
Couldn't agree more. Software engineering is a very practical, fast-moving subject, and that makes most academic work even more irrelevant - software engineering academics often don't have the first clue about what real software engineering is about, because they don't have recent industrial experience.

ChrisMarshallNY 6y ago

I find this amusing. Know how much a Honda Accord costs? About 25 Grand. Know how much a Mercedes S450 costs? About three times as much. They are both great cars, that will be rewarding to own. The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface. The reason is that all those little things that go into high quality take lots of time. I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot. However, if you want that app to be ship quality, and highly usable, you're looking at over a month. The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.

darkerside 6y ago
Your base assumption seems to include that quality is valuable for its own sake. I don't totally disagree, but I'm wary of assigning value based on effort rather than output. Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.
- ChrisMarshallNY 6y ago
  You are exactly correct. That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale. I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger." Our prototypes were incredibly expensive. A $2,000 (retail) body prototype would be insured at half a million bucks. I just find, in my experience, managers expect Mercedes-level quality, for Kia (not even Honda) prices, and (if you are lucky) mature Honda assembly line development speed. That can lead to real Jurassic-scale disasters. If you want that level of quality, you need to plan for it. I'm working on the app that I'm doing, because I was helping a friend of mine evaluate contractors for his dream. The promises they made were absurd. After a couple of these, I just said "Screw it. I'll do it for you." He'll be getting Mercedes quality, but he'll need to wait a bit longer for it. I am pretty good at doing damn good; damn fast. But he also won't be paying a dime for it, so I think he's good with that.

Netcob 6y ago

Integrations for time-tracking software with issue tracking software exists, but I've never actually seen them used in combination before. You'd think that issue trackers would always include a very user-friendly time tracker by default, and advertise integrations with popular time trackers just to get that "actual time spent" data. I've also never looked at a burndown chart with any particular interest after knowing you can close an 8h issue after .5h and a .5h issue after 8h of work and the issue tracker will just keep pretending its charts mean anything. Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.

458aperta 6y ago

I disagree with the premise of this article. Software development is largely a waterfall process even if you use TDD (which was using a machine gun to chop down a tree) and the effort estimation is both predictable and consistent. At least with the model that we use in-house developed over the past 10 years that has resulted in several exits. Don't rely on HN submissions to shape your view of the world. Pedantry, pendants waste your time. Only real world results and data matter. The people who can't do simply teach or publish papers so all you are left are people who grab a leg or an arm and think its an elephant in complete darkness. You simply will never ever find us publishing these information nor do others that have figured it out. Transparency when it comes to winning trust is a must. Disclosures in the name of generosity or some collective good is lust; You might feel good but it ultimately results in increased competition and higher operating cost until your edge disappears. We are not in the business of helping our competitors, we use our unique edge to push them out of the market when we can, we use capital to buy them out when we can't.

alisaus6 6y ago

Hottie hangout pics with nude babes - https://adultlove.life https://adultlove.life

gpmcadam 6y ago

Cache/mirror http://webcache.googleusercontent.com/search?q=cache:http://shape-of-code.coding-guidelines.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/&client=safari&hl=en-gb&prmd=ivn&strip=0&vwsrc=0 http://webcache.googleusercontent.com/search?q=cache:http://...

Londane 6y ago

The site is down for me, so : https://web.archive.org/web/20210118200327/http://shape-of-code.coding-guidelines.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/ https://web.archive.org/web/20210118200327/http://shape-of-c...

distalx 6y ago

Broken Link....

ThomPete 6y ago

In his book “Code Complete”, Steven McConnel speaks about metaphors. He reasons that metaphors are necessary to be a good developer as it helps visualize the act of coding. The metaphor he prefers is the “construction” metaphor. This metaphor he argues best explain the act (some would say art) of programming and gives developers a language to speak in that brings clarity to the development process. When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc) The problem with that metaphor IMO is that it's not actually what software development is. The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building. And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does. I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.

randomsearch 6y ago
The reality is that software development is nothing like engineering or construction, it's totally different. You don't build a quick house, let people live in it and start building the walls whilst they live there. Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.
- ThomPete 6y ago
  My point was more that the software developer have to not just build with existing material and equipment but have to invent and build those on the fly.

SKILNER 6y ago

Not only do we not know how to predict how long a software project will take, we don't even know how to predict what the end product will look like. So who are we kidding? Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team. About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.

xpe 6y ago
Excellent point. Can you share your presentation or at least some of your thinking behind it? Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).

jacques_chester 6y ago

I think a better title would have been "most effort estimation literature relies on small datasets". The use of "fake" can be read as insinuating deliberate malpractice.

dboreham 6y ago

Redirect loop on TFA? Update: working now.

Kaze404 6y ago

I had a conversation about estimates during a recent interview. I asked about how the company deals with those, and the interviewer said they don't do estimates because there's never been a time where something productive came out of one, and I think it makes sense. In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.

snidane 6y ago
When a company uses software estimation, it suggests strong distrust towards the software people and is looking for justification of those huge costs. Most often it means some shitshow happened or is still going on in there.

yters 6y ago

It is provably impossible to predict development effort.

k__ 6y ago

I think, software development works best without deadlines, but then you have to find ways to finance yourself while people wait.

didibus 6y ago

The issue with estimates are expectations. While nobody acknowledges it, you're not actually asked for an estimate, you're being asked for a quote. The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business. If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team. Now keep in mind that the "business" in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the "business" will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.

quickthrower2 6y ago
That is the degenerate scenario, but it’s not always true. Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it. And I’ve also seen wise use of estimation but that is rarer!
- didibus 6y ago
  > And I’ve also seen wise use of estimation but that is rarer I've got a question for you here, what is a wise use of estimation? What problems are estimates the ideal solution for? In the context of software development? I can think of only two: planning headcount and priorities (aka planning the order in which you do things). When you focus on those problems, you realize there's no need for an accurate estimate. All you need is relative sizing between requests. This seems twice as much work as that but gives us half the benefit. Ok probably we shouldn't do this one first. And even then, often time impact is all you need for priorities. Like who cares what takes you longer, if you just focus on building the next most impactful thing you'll very often succeed. The question of: "When are we going to be done this?" Is almost always irrelevant except for what I mentioned "asking for a quote". Like, who cares when it's done? What else you'd rather we do? I can tell you if your alternative will be any faster or not. And is this not the most impactful thing? If so, let's finish it and stop slowing us down with these irrelevant questions. Now for headcount, it's the same thing. Look how much you did last year with what headcount. Ask yourself if what you're hoping to do this year is of the same relative size, twice as big, half as big, or smaller? And then plan headcount accordingly. If twice as big, double head count. Overtime you'll learn if twice as big requires doubling headcount or 1.5x or 1.2x or 2.3x etc.

mr_tristan 6y ago

I wish as much attention was paid to perform post mortems regularly then to only do estimation. You know, actually look at "hey, this is what we guessed and this is what actually happened". I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?" So clearly, there's value, at least when we've done them. I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".

mr_tristan 6y ago
To tie this a bit more into the actual article: It might be even more accurate to just ask people in a post mortem for some feedback, instead of trying to build some data set based on estimates and SLOC. Like an exit poll.

taeric 6y ago

My problem with this is all estimation is hard. Period. Quantitative discussions of something that has been done? Fairly easy, if still not accurate. Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done. Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?

edelans 6y ago
I like this analogy. But I was picturing myself asking my sales manager "How long would it take you to learn to play a song on piano?", and I'm pretty sure he would reply "but you touched a piano before, you are supposed to be a professional piano player ! A professional piano player surely knows how long it would take him to learn how to play a song". So I guess he would miss the point totally :/
- taeric 6y ago
  Ironically missing that some folks will never be able to learn some songs. And, learning to play an existing song doesn't necessarily translate to being able to write one.
perl4ever 6y ago
Estimation is hard if you are stuck in the mindset that you have to make a point estimate. It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that. It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know. It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.
- valenterry 6y ago
  This! If you estimate, estimate a distribution.
- 6y ago

vincentmarle 6y ago

I've been using the Rumsfeld Matrix to classify software estimations into 4 categories: Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments). Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code). Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity). Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.

djoldman 6y ago

>Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression. >...non-machine learning techniques, e.g., regression. Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.

disgruntledphd2 6y ago
They (almost certainly) mean linear regression, which many people appear to regard as statistics and boring, as opposed to single layer neural networks, which are cool. To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).

MattGaiser 6y ago

Estimation seems like a paperwork generation exercise anyway, so why not take it all the way to research?

meesles 6y ago

It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates. According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather. If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.

randomsearch 6y ago
I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.
- deleted 6y ago
  [deleted]
- avdlinde 6y ago
  Well, the metric is #papers and #citations to get funding. So yes that's true, ideally, but...
  - JamesBarney 6y ago
    Relevant blog post - It's not the incentives it's you. https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...
    - angry_octet 6y ago
      Could not disagree more. The premise of this piece is that dieting is just willpower, beating addiction is the same, an entirely individual problem. The reality is that the incentives are what society is asking for, with ethics etc acting as constraints only, rather than actively rewarded. In that formulation it is inevitable (and optimal) that some people will skate to the edge, and if the edge is poorly enforced they will increasingly go over. Policies must address the overall actual effects, not just a chimerical ideal. (It is even more ridiculous coming from someone in psychology.)

lugu 6y ago

I bet it is possible to give precise estimates, the problem is the amount of time and effort to produce such estimation compares with the actual execution.

gregors 6y ago

It's worth watching Software Schedules by Joel Spolsky has some of the best insight regarding evidence based scheduling. It's extremely interesting - https://www.youtube.com/watch?v=EUS4ktQJOSY https://www.youtube.com/watch?v=EUS4ktQJOSY

Groxx 6y ago

One important thing I've managed to get a couple managers to track for quarter/half planning purposes: uncertainty. Prior to that, we'd of course hemmed and hawed and verbalized the fuzziness of our estimates... but only the final decided-on not-super-pessimistic number was written down, and decisions were made based on those. That was a major source of our estimating problems. Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are. It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.

samsk 6y ago

My bulletproof formula for doing estimations in corporate environments: InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order Example: 1 day (MD) = 2 weeks 2 weeks = 4 months

kylecordes 6y ago

Sometimes a request for an “estimate” is really a request for a promise, quotation, a guarantee that something will be delivered by X time or cost. It's easy to detect this: Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%? If you get emotional pushback to discussing uncertainty, they are looking for a promise.

xpe 6y ago
Yes, many of us are too likely to interpret the word "estimate" at face value. It is a wonderful idea to view this instead as only a starting point -- an information-gathering conversation -- as how to provide your customer or other stake-holders what they need. If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)

yibg 6y ago

To me the difficulty of estimating software projects isn't about estimating the time / effort to build the thing. The biggest uncertainty is trying to estimate what it is we're trying to build in the first place. i.e. the details of the spec. Often times where I've seen estimates really off is when the scope changes, or more details are discovered etc. To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.

deandree_ 6y ago

Exactly, you can't really predict what can't be predicted

BlargMcLarg 6y ago

I seriously don't understand the obsession with estimates in the paradigm it is pushed in. Want to be steady at a reliable pace, work for a few months and figure out the average and deviations. Need a priority on tasks, use a priority queue using whatever method you assign it by, often a combination of deadline, value added, complexity and more. Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models? Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.

Jweb_Guru 6y ago
That is absolutely what it is. For example, people are comparing favorably to how sales reps are now forced to record everything they do in something like Salesforce to try to prove they are productive, which by all accounts is pretty awful for the actual people doing the selling and doesn't actually improve sales--it just makes management feel like they're doing something.

snidane 6y ago

Software development which is a repeatable and already defined process is totally possible to predict and estimate. Most tasks of repeatable processes follow normal distribution and is predictable. Deviations from expected mean will be due to predictable factors of the environment such as failed disk or sleepy programmer. You can apply arbitrary six sigma methodology to measure such process with accuracy. The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another. Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts. Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance. Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity. Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.

k__ 6y ago
"a repeatable process would be immediately automated away by writing a function, library, framework or any such tools" This is the crucial point here. Source code is a (almost) self-assembling blueprint. The actual product that will be build is the software and software is a configuration of matter, in this case of a computer. The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house. With better robotics, we will probably get there at some point in the future. And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.

encoderer 6y ago

Don't do an estimate, build a prototype. Don't do an estimate, build a prototype. Don't do an estimate... seriously... build a prototype. And then throw it out. And THEN do an estimate. It will probably be pretty accurate. Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.

mcguire 6y ago

Would software effort estimation work better if software developers were held to fixed-price bids?

tobyhinloopen 6y ago

I feel like I can get pretty good estimates on the following conditions: - the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.

commandlinefan 6y ago
> the application is thoroughly specced. You might need WEEKS for this More weeks than it will end up taking to build the finished product, in fact.

mxcrossb 6y ago

I need to get a check list for the ever so popular criticisms of academia article. This one surely would hit plenty of boxes: scientists don’t pay attention to industry, researchers only care about grants, etc. It’s a lot of words just to say “they need bigger datasets”, which is also a check box. The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.

rietta 6y ago

I've been in this business long enough to know that point estimates are always wrong. A proper estimate is a range with a confidence interval. When forced to do a fixed bid, you have to raise the price even higher to the upper end of the cone of uncertainty.

laichzeit0 6y ago
This agrees with my experience as well. Stop giving point estimates.

sn_master 6y ago

That's how I always felt about all the estimating parts in CS courses. They were all made up BS that would only work on very pre-defined things, like creating a new DB or website based on a template. aka "IT" and not "SE" stuff.

perl4ever 6y ago

I had access to information about historical projects, so I compared the actual amount of time taken to the estimated time at the beginning, for every software project in the history of my organization. I found that on average, things take twice as long as expected. So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it. A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot. But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.

commandlinefan 6y ago
Hofstatder's law: It always takes longer than you expect, even after accounting for Hofstatder's law.

jugg1es 6y ago

I think there is a big difference between building something totally new versus a modification to an existing system. If you are building something totally new, I do not think there is much value in estimates with granularity lower than a month.

deleted 6y ago

[deleted]

franzwong 6y ago

In my previous company, we needed to put project code into timesheet for every activity. Of course, requirement gathering is also an activity. However, before you get the budget, you don't have the project code. Also, you can't get the budget before you have the estimate. It means I needed to give an estimate before I know what system I am going to build.

valenterry 6y ago
How about getting a budget to do an estimation/PoC/prototype for a project?

deleted 6y ago

[deleted]

morelandjs 6y ago

My issue with effort estimates is that it is taboo to report estimates with uncertainty (think how silly that sounds). I'll often give a low-end / high-end and the person asking for the estimate will tell me they want a single number. If you did this in physics you would get laughed and for good reason. It's bad science, and it's bad forecasting. You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?

newscracker 6y ago

So research into software estimation also seems to follow Hofstadter's Law: “It always takes longer than you expect, even when you take into account Hofstadter's Law.”

ChicagoDave 6y ago

Unfortunately, the business folks insist on making up arbitrary dates and costs. It’s the way of the world.

galaxyLogic 6y ago

I think the problem is that if you have a timeline you can produce the result within time and budget, but at the expense of quality. The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).

grey-area 6y ago

It is possible to deliver most software on time, but you need to be very clear about scope and unknowns and keep the units of work small. The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools). If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation. When new ideas arise, they should usually be kept for future work. When work is added, other work needs to be removed or the estimate revised. If the work runs late, something should be removed from the scope to keep it on time, then done after delivery. If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.

toolslive 6y ago

Hasn't anyone tried to replace time estimates with probabilities? (Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)

rightbyte 6y ago
I don't know if I am supposed to roll a 20 on a D20 or a 8 on a D8 to begin with. Time estimation on a macro or micro level is just educated feelings. The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice. I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.

parentheses 6y ago

The first key mistake of software estimation in industry is that we rarely talk about confidence (e.g “I’m 50% confident it’ll be shipped in 5 hours, but 99% confident it’ll ship before end of week”). The fact that this is not part of planning means it’s often not part of the conversation and that we seldom think about estimation as a tuple of value and confidence. The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.

zomars 6y ago

Jonathan Stark saved my soul from this estimates nightmare. You should check him out!