6 ms·

Coding, Fast and Slow: Developers and the Psychology of Overconfidence (2013)

204 points by sidcool 12y ago

jgamman 12y ago
i read this as a break from writing a functional spec. And i've already read Thinking Fast and Slow. And I know that I'm going to be wrong. And I know that no matter what i say to the client, they will say 'OK' and then wonder why i missed the estimate. And i also know that this is what normal looks like and when i'm in the middle of it and it seems as though it's a spaghetti diagram of dependencies, it will work itself out so long as i keep chipping away at it. this is a strange profession - i wouldn't build a bridge like this...
BrandonBradley 12y ago
'The key is that you first accept that making accurate long-term estimates is fundamentally impossible.' My life has been much better since I accepted this. Thanks for the reminder.
bmj 12y ago
Previous discussion here: https://news.ycombinator.com/item?id=5596578 https://news.ycombinator.com/item?id=5596578
jonifico 12y ago
The inaccuracy of your plan is equal to the amount of things you planned in the first place. It's a lot more important to focus on a goal and react according to what the journey throws at you.
deleted 12y ago
[deleted]
igammarays 12y ago
This is why I've learned to always under promise and (try to) over deliver. Whenever I think a project may take me 4 hours, I tell them 3 days. 1 week = 1 month, and so forth. Another valuable technique is the Pomodoro one, dividing one's work into intense 25 minute chunks of time. One then starts to think about work in terms of number of Pomodoros, giving oneself better a better idea of quantities of work. There are other productivity and health benefits as well.
falcor84 12y ago
The obligatory reference to Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. https://en.wikipedia.org/wiki/Hofstadter's_law https://en.wikipedia.org/wiki/Hofstadter's_law
padobson 12y ago
Every project has 'gotchas'. And I don't mean just every coding project, I mean every project anywhere, ever. Part of the key of navigating those 'gotchas' is to immediately communicate it to your stakeholders when you run into one. Sit down and watch HGTV for a an hour or two, and you'll see that carpenters, plumbers, electricians, and HVAC contractors all run into problems - 'Whoops, this wall is load bearing, we'll need a beam here and it's going to take us another 3 days to add it.' 'Your main drain is made out of clay, and we'll need to spend 25% of your budget to replace it, looks like you won't be getting that powder room after all' 'The main of your HVAC stack is right where you want to put that doorway, we'll have to spend another $5k on this project to re-route it.' People that typically finance projects are used to hearing these kinds of problems. Don't assume they won't like hearing them and will fire you - that's a huge mistake. Be honest and upfront with them. Problems like these are not your fault, just be clear on what it's going to take to fix them.
- jiggy2011 12y ago
  The difference between programming and construction projects is that property owners have usually already invested huge amounts of money into the property, so an extra few thousand doesn't seem like a huge deal and the extra work is often financed as part of the mortgage. Construction contractors also don't promise "magic" like programmers do.
  - padobson 12y ago
    A few things: 80% of the projects I've worked on already had code written for them when I showed up to work on them. There's no difference between the existing investment in legacy code and the existing investment in a property. New home builds also have gotchas. Programmer's promising "magic" goes back to my original point of communicating with the stakeholders. If you're promising "magic", you're just asking for the project to fail. Be clear an upfront, and your customer will realize you're an engineer and not a miracle worker.
    - jiggy2011 12y ago
      Property investment is a little different because you can usually recoup at least some of the money spent at a later date by selling the property. A building without a wood rot problem will always be worth more than one with one, especially when the problem is likely to get worse over time if it's not dealt with. A software project on the other hand is typically worth close to nothing if it doesn't fulfil it's requirements , so the sunk costs are different. To the second point, the problem with programming projects is that you are often competing against more enthusiastic and less experienced devs who are naive and under estimate.
ThomaszKrueger 12y ago
I usually give "them" a shocker estimate, and my rationale is pretty simple - unknowns. And for the most part my estimates are usually right. If asked to trim it, I simply state that OK, I'll deliver what may not be the right product but then it will become a matter of maintenance instead of development. Your choice.
rayiner 12y ago
One of the best things you can do in your organization is create a culture that treats obstacle discovery as forward progress. There is nothing more disheartening for an engineer than banging her head against the wall trying to debug some race condition only to get flak from management about why it isn't fixed and why it's holding up some milestone. The root of this problem is management that doesn't understand the problem domain, and is suspicious about whether the engineers are really working diligently. The proper response to obstacle discovery is to treat it as forward progress. Firing off an email to a superior saying: "I discovered an unknown race condition, let's see what we need to do to fix it," should be treated no differently than "I finished this feature." The presumption should be that the engineer used due diligence and the situation was unforeseen. Now, that doesn't preclude separate improvements to diligence procedures to provide better foresight in the, future but under no circumstances should anyone be blamed for discovering obstacles or defects and raising them. When your culture does that, all you'll get is shitty product.
- chicofarm 12y ago
  Interesting. This sounds like an Agency Theory issue. http://www.investopedia.com/terms/a/agencytheory.asp http://www.investopedia.com/terms/a/agencytheory.asp
- lucaswoj 12y ago
  Are you hiring? Can I work for you?
- neebz 12y ago
  forward progress means learning. if our colleague gets into an unexpected issue and he takes time to solve, we pat him for it. however it's important that the learnings have to be shared. they should be tangible enough to be reapplied if we get into a similar situation in future.
- markbnj 12y ago
  Dead on, imo.
- sporkenfang 12y ago
  Having experienced both sides of this, I concur. When your manager is annoyed and replies "oh man, how much will I have to push back the next release deadline for your bug?!" even though you just discovered it and didn't necessarily cause it, it's much more disheartening than something like "Good! Fix it! I'm giving you two weeks until the next release (instead of releasing tomorrow) and if it's still an issue we'll get you some backup." Race conditions can in some cases break the entire product or render some of its results useless (depending on scale and location of the bug, naturally). I definitely think in most cases they shouldn't be taken lightly.
kator 12y ago
I've been developing for 30 years. When asked for an estimate I spend time thinking about it and then when I come up with a number I multiply it by 3. Usually this works but sometimes I can be off. My typical process is to score stuff from "totally get it" to "don't have it all in my head". I then guess on hours and in the end multiply by this 3 factor. When my clients complain about the estimate I often say we "write software": 1) The way the client thought it should be written 2) Again the way the developer thought it should be written 3) And finally the way is should have been written in the first place I don't remember where I picked this up at, it was eons ago, but the axiom seems to work. When my teams give me estimates I almost always fudge by this 3x factor when working with my stakeholders and explain how hard it is to really estimate development without wasting tons of time doing massive waterfall charts. Then they ask for the chart and I laugh "but then it'll be wrong two seconds after we publish it." In all my years of coding there have been "wow that went fast" times on projects and "oh man I'm dying here" and the more room I leave to reduce stress in the "dying" phase the easier it is to break free and get back to "fast" mode.
- d--b 12y ago
  It's exactly the same thing I have been doing. This is a common project management practice. If you translate this practice in the terms of the original article. This is a kind of double thinking. You know the time it's going to take you, but you also know that you need to multiply it by 3 for it to be correct.
- anthay 12y ago
  But you risk your manager (if you have one) telling you you are "sandbagging." And you might see others on your team tell your manager "it's no big deal - an hour or so of work". And they slap some crap in there that whacks the mole down long enough for the manager to make a mental note to give your team mate a bigger raise than you.
  - Ntrails 12y ago
    If you want shitty work that creates a mountain of technical debt, I can do that (no, really, I'm a natural at it). If you want a thought through process that covers all of the points as best I can, I will do that. Align my incentives. But don't complain because I do the optimal thing
marktangotango 12y ago
>> if you were to write a specification in such detail that it would capture those issues, you’d be writing the software This is the key insight imo.
rectang 12y ago
A rule I learned from my father: Multiply any software estimate by pi.
paulhauggis 12y ago
When management has no development experience, it's even more impossible. At pretty much every place I've ever worked, features get added and the scope changes many times before the project is finished. When it's not finished on time, management likes to blame the developers and can't figure out why nothing is finished on time.
svec 12y ago
I've enjoyed their blog, especially "Speeding Up Your Engineering Org, Part I: Beyond the Cost Center Mentality". If the author(s) are reading this, when will Part II come out???
diltonm 12y ago
It's an excellent article, at least three different times I caught myself shaking my head in agreement. I really liked, "Congratulations, you’ve just invented Waterfall.".
amelius 12y ago
One unaddressed problem with estimates is: if the developers finish before the project is due according to the estimate, there is no motivation to start doing new work. How to solve this?
- t0rped0 12y ago
  While it only happens on a rare occasion, whenever I finish my projects early I'm always excited to show the client/management that we've been able to make better progress. However, I'm extremely cautious not to let their excitement alter their perspective about future estimates being completed early. Another issue is that often clients/management will jump the gun into the next phase after an early completion without allowing the devs a much needed break. Perhaps this is one of the fears that keeps devs silent when (if ever) they finish early, or stretching their work till the end of the original estimate.
- saraid216 12y ago
  Add more tests.
- jacalata 12y ago
  Either get better developers, or get better management. Usually the second, because if your management doesn't know how to reward your devs for good work then they are going to cause more problems than just this one.
tedks 12y ago
It is possible to become better calibrated for long-term predictions. This takes a longer amount of time, but is still doable. It's an intuitive process, and the way I do it relies on discovering the true probabilities associated with various feelings of confidence I have in a given domain. The best tool I've yet found to do this is PredictionBook.com. It could use things like tags support (so you could see your accuracy for sports separately from your accuracy for development), but it's still useful overall. I've gotten much better at being accurate, especially for 90% predictions, which is a useful benchmark.
nevergetenglish 12y ago
To estimate the time required to design your system you need to estimate the variance of the time required. A well known result is that the variance of a sum of independent variables associated to independent problems is the sum of the variances of the subproblems. What this mean in practice? It means that you should try to decompose your problem in independent and dependent subproblems (perhaps associated to features). Independent subproblems their variance, the formula for dependent problems depends of the correlation between them. What I am trying to say is that if you know a little math and have an intuitive knowledge of the structure of the problem divided in sub-problems you could make a much better estimate of the variance (and this mean there is much less uncertainty in the delivery time). One could design a program that constructs a graph and a knowledge base of previous programs that you have designed. The graph edges should be weighted in accordance with the correlation factor of dependency of those components. Now that I am writing this, it seems very likely that someone has implemented such a scheme to estimate time delivery. Perhaps there are start-ups using this system to estimate delivery time, since all this is basic math. Edited: Mostly grammar.
michaelochurch 12y ago
I tend to have a darker, more cynical view. Asking for an estimate is often a power play or "microaggression". It's more often about showing dominance than any legitimate business need to know this unknowable quantity. It plays a programmer's present self against her future self. Her future self wants an accurate or even pessimistic estimate to be made, so there are no unpleasant surprises to management and painful conversations resulting from inflated expectations. But her present self wants the guy standing at her desk to go away so she can get back to work. An unreasonably optimistic estimate gives the powerful, annoying people what they want so they go away. A realistic estimate is going to lead to obnoxious follow-on questions. "Why's it going to take so long?" "I don't know, but experience leads me to think..." "But you see no specific reason why this can't be done in 2 weeks?" "Well, no, because there are unknown unknowns..." "Great! Two weeks!" This degenerates for two reasons. First, managers tend to believe that, even if overly optimistic estimates are more inaccurate, the work is done more quickly with unreasonable estimates (which become "milestones", then "deadlines"). So they take this as an incentive to be more irritating because it "makes people work faster". Second, it leads to poor planning and brittle schedules and much more undesirable variance. Obviously, the good software managers don't play these games, but they're probably 1 in 10. Because there is so much money in software, and because programmers refuse to organize and allow themselves to be underpaid, that creates lots of room for mismanagement. Developers are also to blame for some of this; because they've been in mismanaged environments for years, many of them have developed a distrust of management that has led them to miscommunicate and (inappropriately) simplify, hence the "estimates" that evolve into deadlines. Overconfidence and miscommunication are, for sure, substantial components.
- Roboprog 12y ago
  Sometimes there are real deadlines, such as for regulatory changes. But, yeah, often it's just bullying to drive a bad bargain. Most management HATES iterative development, because it's too honest. One of my favorite Ed Yourdon quotes: "Vote with your feet" (from "Decline and Fall of the American Programmer, which scared the hell out of me, rightly so in some cases - and after working with off shore development, that scared the hell out of me)
kazinator 12y ago
> “We should just be more careful at the specification stage”. But this turns out to fail, badly. Why? The core reason is that, as you can see from the examples above, if you were to write a specification in such detail that it would capture those issues, you’d be writing the software. Not true; a specification can precisely describe the properties of the solution of a (sub-)problem, rather than the process of solving it.
- JoeAltmaier 12y ago
  Its an iterative process that approaches code. The more abstract the language, the closer the spec resembles the code.
  - kazinator 12y ago
    The closer are the abstractions in the coding language to those in the specification language, the closer the spec resembles the code. If the two abstract in different ways, then not.
- Roboprog 12y ago
  You must have much smarter users reading your requirements than I do. But seriously, though, YMMV, and the mileage on this one is all too often very low. If the requirements are detailed enough to be accurate, the users don't understand them, or simply agree to blatant inaccuracies. Iterative development really is The Way, but The Management simply doesn't want to hear it. "What part of 'Loser' didn't you understand?" said the Sociopath (as long as hiding requirements = free extra labor).
wazoox 12y ago
Indispensable reference to Joel: http://www.joelonsoftware.com/items/2007/10/26.html http://www.joelonsoftware.com/items/2007/10/26.html
jawngee 12y ago
Interestingly, I seem to have the inverse problem. People telling me how easy it is. I don't do web dev as a matter of course these days (I did nothing but back in the day though), but once in awhile I help a friend out with their portfolio of sites. Usually easy updates. But the conversation almost inevitably starts out with, "Do you think you could take care of it? It should take a few hours". Of course, it takes two days.
dschiptsov 12y ago
The old maxim goes like 10 lines per day on average (of fully "developed" and tested code). 90% of time should be spent on studying textbooks, reading other people's code (only the best authors, like Joe Armstrong or Simon Marlow or Rich Hickey, you know), [wishful] thinking, visualizing, drawing diagrams (using a pen and paper, no UML and shit), writing pseudo-code (like they do in AIMA), [unit]tests (before code!) and then just 10% of time for coding, when you know what exactly you are doing and why. This, by the way, is called "slow thinking".)
Animats 12y ago
Why does programming have this problem? No unions. Really. Compare film scheduling and construction scheduling, which are well developed disciplines. That's because, in those industries, there's paid overtime, and at rates higher than straight time. Thus, "crunches" caused by overoptimistic scheduling add labor costs, which come right out of the company's profit. This forces scheduling discipline on management. There's a tendency to overestimate, rather than underestimate.
lordnacho 12y ago
One thing I didn't see mentioned is moving goalposts. If you think something is easy, you're more likely to accept adding some bells and whistles. So it will take at least the time you set aside for it. Or more.

204 points by sidcool 12y ago

i read this as a break from writing a functional spec. And i've already read Thinking Fast and Slow. And I know that I'm going to be wrong. And I know that no matter what i say to the client, they will say 'OK' and then wonder why i missed the estimate. And i also know that this is what normal looks like and when i'm in the middle of it and it seems as though it's a spaghetti diagram of dependencies, it will work itself out so long as i keep chipping away at it. this is a strange profession - i wouldn't build a bridge like this...

BrandonBradley 12y ago

'The key is that you first accept that making accurate long-term estimates is fundamentally impossible.' My life has been much better since I accepted this. Thanks for the reminder.

bmj 12y ago

Previous discussion here: https://news.ycombinator.com/item?id=5596578 https://news.ycombinator.com/item?id=5596578

jonifico 12y ago

The inaccuracy of your plan is equal to the amount of things you planned in the first place. It's a lot more important to focus on a goal and react according to what the journey throws at you.

deleted 12y ago

[deleted]

igammarays 12y ago

This is why I've learned to always under promise and (try to) over deliver. Whenever I think a project may take me 4 hours, I tell them 3 days. 1 week = 1 month, and so forth. Another valuable technique is the Pomodoro one, dividing one's work into intense 25 minute chunks of time. One then starts to think about work in terms of number of Pomodoros, giving oneself better a better idea of quantities of work. There are other productivity and health benefits as well.

falcor84 12y ago

The obligatory reference to Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. https://en.wikipedia.org/wiki/Hofstadter's_law https://en.wikipedia.org/wiki/Hofstadter's_law

padobson 12y ago

Every project has 'gotchas'. And I don't mean just every coding project, I mean every project anywhere, ever. Part of the key of navigating those 'gotchas' is to immediately communicate it to your stakeholders when you run into one. Sit down and watch HGTV for a an hour or two, and you'll see that carpenters, plumbers, electricians, and HVAC contractors all run into problems - 'Whoops, this wall is load bearing, we'll need a beam here and it's going to take us another 3 days to add it.' 'Your main drain is made out of clay, and we'll need to spend 25% of your budget to replace it, looks like you won't be getting that powder room after all' 'The main of your HVAC stack is right where you want to put that doorway, we'll have to spend another $5k on this project to re-route it.' People that typically finance projects are used to hearing these kinds of problems. Don't assume they won't like hearing them and will fire you - that's a huge mistake. Be honest and upfront with them. Problems like these are not your fault, just be clear on what it's going to take to fix them.

jiggy2011 12y ago
The difference between programming and construction projects is that property owners have usually already invested huge amounts of money into the property, so an extra few thousand doesn't seem like a huge deal and the extra work is often financed as part of the mortgage. Construction contractors also don't promise "magic" like programmers do.
- padobson 12y ago
  A few things: 80% of the projects I've worked on already had code written for them when I showed up to work on them. There's no difference between the existing investment in legacy code and the existing investment in a property. New home builds also have gotchas. Programmer's promising "magic" goes back to my original point of communicating with the stakeholders. If you're promising "magic", you're just asking for the project to fail. Be clear an upfront, and your customer will realize you're an engineer and not a miracle worker.
  - jiggy2011 12y ago
    Property investment is a little different because you can usually recoup at least some of the money spent at a later date by selling the property. A building without a wood rot problem will always be worth more than one with one, especially when the problem is likely to get worse over time if it's not dealt with. A software project on the other hand is typically worth close to nothing if it doesn't fulfil it's requirements , so the sunk costs are different. To the second point, the problem with programming projects is that you are often competing against more enthusiastic and less experienced devs who are naive and under estimate.

ThomaszKrueger 12y ago

I usually give "them" a shocker estimate, and my rationale is pretty simple - unknowns. And for the most part my estimates are usually right. If asked to trim it, I simply state that OK, I'll deliver what may not be the right product but then it will become a matter of maintenance instead of development. Your choice.

rayiner 12y ago

One of the best things you can do in your organization is create a culture that treats obstacle discovery as forward progress. There is nothing more disheartening for an engineer than banging her head against the wall trying to debug some race condition only to get flak from management about why it isn't fixed and why it's holding up some milestone. The root of this problem is management that doesn't understand the problem domain, and is suspicious about whether the engineers are really working diligently. The proper response to obstacle discovery is to treat it as forward progress. Firing off an email to a superior saying: "I discovered an unknown race condition, let's see what we need to do to fix it," should be treated no differently than "I finished this feature." The presumption should be that the engineer used due diligence and the situation was unforeseen. Now, that doesn't preclude separate improvements to diligence procedures to provide better foresight in the, future but under no circumstances should anyone be blamed for discovering obstacles or defects and raising them. When your culture does that, all you'll get is shitty product.

chicofarm 12y ago
Interesting. This sounds like an Agency Theory issue. http://www.investopedia.com/terms/a/agencytheory.asp http://www.investopedia.com/terms/a/agencytheory.asp
lucaswoj 12y ago
Are you hiring? Can I work for you?
neebz 12y ago
forward progress means learning. if our colleague gets into an unexpected issue and he takes time to solve, we pat him for it. however it's important that the learnings have to be shared. they should be tangible enough to be reapplied if we get into a similar situation in future.
markbnj 12y ago
Dead on, imo.
sporkenfang 12y ago
Having experienced both sides of this, I concur. When your manager is annoyed and replies "oh man, how much will I have to push back the next release deadline for your bug?!" even though you just discovered it and didn't necessarily cause it, it's much more disheartening than something like "Good! Fix it! I'm giving you two weeks until the next release (instead of releasing tomorrow) and if it's still an issue we'll get you some backup." Race conditions can in some cases break the entire product or render some of its results useless (depending on scale and location of the bug, naturally). I definitely think in most cases they shouldn't be taken lightly.

kator 12y ago

I've been developing for 30 years. When asked for an estimate I spend time thinking about it and then when I come up with a number I multiply it by 3. Usually this works but sometimes I can be off. My typical process is to score stuff from "totally get it" to "don't have it all in my head". I then guess on hours and in the end multiply by this 3 factor. When my clients complain about the estimate I often say we "write software": 1) The way the client thought it should be written 2) Again the way the developer thought it should be written 3) And finally the way is should have been written in the first place I don't remember where I picked this up at, it was eons ago, but the axiom seems to work. When my teams give me estimates I almost always fudge by this 3x factor when working with my stakeholders and explain how hard it is to really estimate development without wasting tons of time doing massive waterfall charts. Then they ask for the chart and I laugh "but then it'll be wrong two seconds after we publish it." In all my years of coding there have been "wow that went fast" times on projects and "oh man I'm dying here" and the more room I leave to reduce stress in the "dying" phase the easier it is to break free and get back to "fast" mode.

d--b 12y ago
It's exactly the same thing I have been doing. This is a common project management practice. If you translate this practice in the terms of the original article. This is a kind of double thinking. You know the time it's going to take you, but you also know that you need to multiply it by 3 for it to be correct.
anthay 12y ago
But you risk your manager (if you have one) telling you you are "sandbagging." And you might see others on your team tell your manager "it's no big deal - an hour or so of work". And they slap some crap in there that whacks the mole down long enough for the manager to make a mental note to give your team mate a bigger raise than you.
- Ntrails 12y ago
  If you want shitty work that creates a mountain of technical debt, I can do that (no, really, I'm a natural at it). If you want a thought through process that covers all of the points as best I can, I will do that. Align my incentives. But don't complain because I do the optimal thing

marktangotango 12y ago

>> if you were to write a specification in such detail that it would capture those issues, you’d be writing the software This is the key insight imo.

rectang 12y ago

A rule I learned from my father: Multiply any software estimate by pi.

paulhauggis 12y ago

When management has no development experience, it's even more impossible. At pretty much every place I've ever worked, features get added and the scope changes many times before the project is finished. When it's not finished on time, management likes to blame the developers and can't figure out why nothing is finished on time.

svec 12y ago

I've enjoyed their blog, especially "Speeding Up Your Engineering Org, Part I: Beyond the Cost Center Mentality". If the author(s) are reading this, when will Part II come out???

diltonm 12y ago

It's an excellent article, at least three different times I caught myself shaking my head in agreement. I really liked, "Congratulations, you’ve just invented Waterfall.".

amelius 12y ago

One unaddressed problem with estimates is: if the developers finish before the project is due according to the estimate, there is no motivation to start doing new work. How to solve this?

t0rped0 12y ago
While it only happens on a rare occasion, whenever I finish my projects early I'm always excited to show the client/management that we've been able to make better progress. However, I'm extremely cautious not to let their excitement alter their perspective about future estimates being completed early. Another issue is that often clients/management will jump the gun into the next phase after an early completion without allowing the devs a much needed break. Perhaps this is one of the fears that keeps devs silent when (if ever) they finish early, or stretching their work till the end of the original estimate.
saraid216 12y ago
Add more tests.
jacalata 12y ago
Either get better developers, or get better management. Usually the second, because if your management doesn't know how to reward your devs for good work then they are going to cause more problems than just this one.

tedks 12y ago

It is possible to become better calibrated for long-term predictions. This takes a longer amount of time, but is still doable. It's an intuitive process, and the way I do it relies on discovering the true probabilities associated with various feelings of confidence I have in a given domain. The best tool I've yet found to do this is PredictionBook.com. It could use things like tags support (so you could see your accuracy for sports separately from your accuracy for development), but it's still useful overall. I've gotten much better at being accurate, especially for 90% predictions, which is a useful benchmark.

nevergetenglish 12y ago

To estimate the time required to design your system you need to estimate the variance of the time required. A well known result is that the variance of a sum of independent variables associated to independent problems is the sum of the variances of the subproblems. What this mean in practice? It means that you should try to decompose your problem in independent and dependent subproblems (perhaps associated to features). Independent subproblems their variance, the formula for dependent problems depends of the correlation between them. What I am trying to say is that if you know a little math and have an intuitive knowledge of the structure of the problem divided in sub-problems you could make a much better estimate of the variance (and this mean there is much less uncertainty in the delivery time). One could design a program that constructs a graph and a knowledge base of previous programs that you have designed. The graph edges should be weighted in accordance with the correlation factor of dependency of those components. Now that I am writing this, it seems very likely that someone has implemented such a scheme to estimate time delivery. Perhaps there are start-ups using this system to estimate delivery time, since all this is basic math. Edited: Mostly grammar.

michaelochurch 12y ago

I tend to have a darker, more cynical view. Asking for an estimate is often a power play or "microaggression". It's more often about showing dominance than any legitimate business need to know this unknowable quantity. It plays a programmer's present self against her future self. Her future self wants an accurate or even pessimistic estimate to be made, so there are no unpleasant surprises to management and painful conversations resulting from inflated expectations. But her present self wants the guy standing at her desk to go away so she can get back to work. An unreasonably optimistic estimate gives the powerful, annoying people what they want so they go away. A realistic estimate is going to lead to obnoxious follow-on questions. "Why's it going to take so long?" "I don't know, but experience leads me to think..." "But you see no specific reason why this can't be done in 2 weeks?" "Well, no, because there are unknown unknowns..." "Great! Two weeks!" This degenerates for two reasons. First, managers tend to believe that, even if overly optimistic estimates are more inaccurate, the work is done more quickly with unreasonable estimates (which become "milestones", then "deadlines"). So they take this as an incentive to be more irritating because it "makes people work faster". Second, it leads to poor planning and brittle schedules and much more undesirable variance. Obviously, the good software managers don't play these games, but they're probably 1 in 10. Because there is so much money in software, and because programmers refuse to organize and allow themselves to be underpaid, that creates lots of room for mismanagement. Developers are also to blame for some of this; because they've been in mismanaged environments for years, many of them have developed a distrust of management that has led them to miscommunicate and (inappropriately) simplify, hence the "estimates" that evolve into deadlines. Overconfidence and miscommunication are, for sure, substantial components.

Roboprog 12y ago
Sometimes there are real deadlines, such as for regulatory changes. But, yeah, often it's just bullying to drive a bad bargain. Most management HATES iterative development, because it's too honest. One of my favorite Ed Yourdon quotes: "Vote with your feet" (from "Decline and Fall of the American Programmer, which scared the hell out of me, rightly so in some cases - and after working with off shore development, that scared the hell out of me)

kazinator 12y ago

> “We should just be more careful at the specification stage”. But this turns out to fail, badly. Why? The core reason is that, as you can see from the examples above, if you were to write a specification in such detail that it would capture those issues, you’d be writing the software. Not true; a specification can precisely describe the properties of the solution of a (sub-)problem, rather than the process of solving it.

JoeAltmaier 12y ago
Its an iterative process that approaches code. The more abstract the language, the closer the spec resembles the code.
- kazinator 12y ago
  The closer are the abstractions in the coding language to those in the specification language, the closer the spec resembles the code. If the two abstract in different ways, then not.
Roboprog 12y ago
You must have much smarter users reading your requirements than I do. But seriously, though, YMMV, and the mileage on this one is all too often very low. If the requirements are detailed enough to be accurate, the users don't understand them, or simply agree to blatant inaccuracies. Iterative development really is The Way, but The Management simply doesn't want to hear it. "What part of 'Loser' didn't you understand?" said the Sociopath (as long as hiding requirements = free extra labor).

wazoox 12y ago

Indispensable reference to Joel: http://www.joelonsoftware.com/items/2007/10/26.html http://www.joelonsoftware.com/items/2007/10/26.html

jawngee 12y ago

Interestingly, I seem to have the inverse problem. People telling me how easy it is. I don't do web dev as a matter of course these days (I did nothing but back in the day though), but once in awhile I help a friend out with their portfolio of sites. Usually easy updates. But the conversation almost inevitably starts out with, "Do you think you could take care of it? It should take a few hours". Of course, it takes two days.

dschiptsov 12y ago

The old maxim goes like 10 lines per day on average (of fully "developed" and tested code). 90% of time should be spent on studying textbooks, reading other people's code (only the best authors, like Joe Armstrong or Simon Marlow or Rich Hickey, you know), [wishful] thinking, visualizing, drawing diagrams (using a pen and paper, no UML and shit), writing pseudo-code (like they do in AIMA), [unit]tests (before code!) and then just 10% of time for coding, when you know what exactly you are doing and why. This, by the way, is called "slow thinking".)

Animats 12y ago

Why does programming have this problem? No unions. Really. Compare film scheduling and construction scheduling, which are well developed disciplines. That's because, in those industries, there's paid overtime, and at rates higher than straight time. Thus, "crunches" caused by overoptimistic scheduling add labor costs, which come right out of the company's profit. This forces scheduling discipline on management. There's a tendency to overestimate, rather than underestimate.

lordnacho 12y ago

One thing I didn't see mentioned is moving goalposts. If you think something is easy, you're more likely to accept adding some bells and whistles. So it will take at least the time you set aside for it. Or more.