6 ms·

This claim of inescapable novelty has a kernel of truth, but it is being exaggerated beyond any sense of proportion. If everything you have done in your day-to-

This claim of inescapable novelty has a kernel of truth, but it is being exaggerated beyond any sense of proportion. If everything you have done in your day-to-day work has been beyond-estimating original, then either you could out-Knuth Knuth (in which case, kudos to you, and I hope you will have time to write the books someday, but your experience is not generally applicable), or you have been goofing off some of the time. That which is passed off as novelty is often ignorance (not necessarily on the part of the developer.) To continue the retail analogy, software projects are often like the guy who is building a deck on his house, but keeps going back to the hardware store because he made no attempt to think things through, or measure before cutting (note that, in this analogy, the would-be builder is not necessarily playing the role of the developer. In some cases, that might be the hardware store owner's role.)

thom 8y ago
But it's exactly this 'ignorance' that is the major part of every software project. There are unknown unknowns, and there is no way from a standing start to grasp either their number or their nature. I'm happy if people are saying that this exploratory work should be done (in part or in total) before estimating, but in my experience that's just not palatable to most organisations. I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.
- mannykannot 8y ago
  Well, then let's call it what it is, because there are things that can be done to mitigate ignorance to some extent, if there is the will to do so from all parties.
  - Aaargh20318 8y ago
    > there are things that can be done to mitigate ignorance to some extent Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.
    - mannykannot 8y ago
      Firstly you you are still assuming the same gross exaggeration of novelty, and secondly, you can do analysis and planning at a much higher level of abstraction than development. Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.
      - Aaargh20318 8y ago
        > Firstly you you are still assuming the same gross exaggeration of novelty I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance > you can do analysis and planning at a much higher level of abstraction than development Of course, but then you'll get an inaccurate estimate. > Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.
        mannykannot 8y ago
        There is a false dichotomy here between knowing everything and knowing nothing, between perfection and utter chaos. You may only have experienced the latter, but there are places where people know pretty much what they are doing, most of the time.
        AuthorizedCust 8y ago
        > ...there are places where people know pretty much what they are doing, most of the time. It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code. On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.
nickthemagicman 8y ago
It's like building a deck except even the screw hasn't been invented yet. And you think: I'll just create a device that attaches the wood like a big paperclip but then realize after you make it that it's not strong enough and won't scale so you create a nail. But then you realize you may need to reuse the wood so you finally create a screw.that can be removed.
- mannykannot 8y ago
  That analogy might have worked in the 1950s.
chrisbennet 8y ago
You aren’t wrong - for some things. I think it’s a spectrum. For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems. It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard. [1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)
ksk 8y ago
Agree 100%. IMHO, a lot of it has to do with the programmer mentality of wanting to be "challenged" to solve problems one step away from their competency. If you're fresh out of school, implementing your own allocator might be a challenge you take upon to solve a memory constraint, when you can just take what someone else (with more experience) has built. And then senior people insist on designing yet another over engineered platform technology because everything has to be 'future prof' (when in reality it never is and only ends up being performance-proof). How many JS frameworks have we had so far that do the same thing and just repeat mistakes of previous ones? Its like one every year.
lmilcin 8y ago
I agree. There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability. It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.

thom 8y ago

But it's exactly this 'ignorance' that is the major part of every software project. There are unknown unknowns, and there is no way from a standing start to grasp either their number or their nature. I'm happy if people are saying that this exploratory work should be done (in part or in total) before estimating, but in my experience that's just not palatable to most organisations. I envy _hugely_ anyone who builds software that is even 50% the same as what they were working on six months ago.

mannykannot 8y ago
Well, then let's call it what it is, because there are things that can be done to mitigate ignorance to some extent, if there is the will to do so from all parties.
- Aaargh20318 8y ago
  > there are things that can be done to mitigate ignorance to some extent Sure, but all you're doing is moving the uncertainty from the development to the planning phase, so instead of a development phase of unknown length and cost, you end up with a planning phase of unknown length and cost.
  - mannykannot 8y ago
    Firstly you you are still assuming the same gross exaggeration of novelty, and secondly, you can do analysis and planning at a much higher level of abstraction than development. Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction, before doing anything else.
    - Aaargh20318 8y ago
      > Firstly you you are still assuming the same gross exaggeration of novelty I disagree. Even when you're doing mundane everyday stuff you have done a thousand times before, one trivial variation may throw of your estimate completely. Maybe an edge-case in this specific project triggers a bug in a 3rd party's library, and you're losing weeks waiting for the 3rd party to roll out the fix (been there, done that). You just don't know in advance > you can do analysis and planning at a much higher level of abstraction than development Of course, but then you'll get an inaccurate estimate. > Waterfall gave the impression that planning and analysis is costly by insisting it should be carried down to the lowest levels of abstraction That's only necessary if you want an accurate estimate. I think accurate estimates are pipe dreams.
      - mannykannot 8y ago
        There is a false dichotomy here between knowing everything and knowing nothing, between perfection and utter chaos. You may only have experienced the latter, but there are places where people know pretty much what they are doing, most of the time.
        AuthorizedCust 8y ago
        > ...there are places where people know pretty much what they are doing, most of the time. It's called spec programming, where you have programmers slinging code against precise specifications. Common in ERP and regulatory or business compliance-driven code. On the other hand, you have development. That is where you're attacking the less defined problems. Sure, we know the tools in our toolset, how to apply them, and time it takes to apply each tool. Problem is, the time to figure out the series of tools needed and their precise applications burns enormous time, time that could have been spent towards a MVP.

nickthemagicman 8y ago

It's like building a deck except even the screw hasn't been invented yet. And you think: I'll just create a device that attaches the wood like a big paperclip but then realize after you make it that it's not strong enough and won't scale so you create a nail. But then you realize you may need to reuse the wood so you finally create a screw.that can be removed.

mannykannot 8y ago
That analogy might have worked in the 1950s.

chrisbennet 8y ago

You aren’t wrong - for some things. I think it’s a spectrum. For the things I work on, and frankly always have, I spend most of my time in the “R” part of R&D. Now granted, I’ve never worked on classical “business” or CRUD type problems. It’s natural to think that once you have lots of experience, that estimating will be easier. In my experience, I found myself simply tackling harder or unique problems. Instead of getting comfortable, you find yourself mostly operating at the edge of your competence. This is great fun [1] but makes estimating pretty hard. [1] I personally enjoy being on the “black diamond” part of the learning curve. (“Black diamond” = most difficult ski trail)

ksk 8y ago

Agree 100%. IMHO, a lot of it has to do with the programmer mentality of wanting to be "challenged" to solve problems one step away from their competency. If you're fresh out of school, implementing your own allocator might be a challenge you take upon to solve a memory constraint, when you can just take what someone else (with more experience) has built. And then senior people insist on designing yet another over engineered platform technology because everything has to be 'future prof' (when in reality it never is and only ends up being performance-proof). How many JS frameworks have we had so far that do the same thing and just repeat mistakes of previous ones? Its like one every year.

lmilcin 8y ago

I agree. There is also a lot you can do to decrease impact of novelty on the overall schedule and most people I work with do exactly nothing to reduce variability in the process and improve their estimation ability. It is easy to claim every time you write code you are doing something new but, hey, every time they build a building they do something new. Yet buildings (except for extremely ambitious projects) typically can be delivered reliably by reputable developers.