22 ms·

Why does software development take so long?

296 points by cheiVia0 10y ago

scrrr 10y ago
Because it's an art. It's like painting a picture. Good art takes time.
- forgottenacc57 10y ago
  This is in fact the correct answer.
- jacquesm 10y ago
  You're just asking for it: http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm
  - tempodox 10y ago
    One (half-) sentence from that article especially caught my eye: ...computer programmers create artifacts that have to stand up to an objective reality. If we could make this more of a practise than a theory, it might substantially improve the state of things.
  - rkeene2 10y ago
    Reading through the article, and having no way to comment there, I'll comment here: > It is true that both painters and programmers make things, just like a pastry chef makes a wedding cake, or a chicken makes an egg. But nothing about what they make, the purposes it serves, or how they go about doing it is in any way similar. This misses the mark almost entirely, from my understanding. Hackers and painters CREATE something -- it does not exist and then it does. The wording Paul Graham uses is "makers", but it's not what I feel so maybe that is the criticism being expressed in this part of the article ? The purpose of artists who paint and hackers /IS/ the same -- to create. To get what is inside of you out. To have the expression which has no form to take form and be in the world as something that can be shared with another soul. The methods are the same -- extract what is inside of you into concrete form, mold it, remove the bits that are not right, add more bits, move the bits around. Certainly this is not true of computer programmers, which the article refers to hackers as but this is a failure of the article not of the original comparison. Nor is it true of all painters, since the word is too generic, which is a failure of Paul Graham's choice of wording for this comparison since based on the rest of the reading that is what he is referring to. Computer programmers program computers, perhaps soullessly. Painters paint things, perhaps soullessly. Hackers hack because they need to. Artists that paint paint because they need to. As to the rest, I don't know if Paul Graham's intention was to attempt to borrow "coolness" from artists, but for me it is not. It's an attempt to explain why one must hack and why that is an identity and not an activity or a profession.
rogual 10y ago
And this is just for a sole developer who knows what he's doing. When you have a team of people at varying levels of ability, a meeting for every decision, JIRA-wielding project managers with burndown charts to keep happy, standards & practices, legal, translators, UI designers, all regularly taking out exclusive locks on each other's time... it's a wonder anything gets done at all.
- hex13 10y ago
  It's correct. I've noticed that when I work in a team I work with maybe 10% of my solo efficiency. Just because of this whole team work bullshit and because of legacy code (it takes me often more time to understand how code written by someone works than it would take to write such code from scratch).
pasta 10y ago
The 'readable' button in Firefox was not working for me. Maybe others also would like to paste this in the url bar: javascript:document.body.style.width='600px';document.body.style.margin='0 auto';void(0);
- StavrosK 10y ago
  Firefox is wonky with its "readable view" button, huh? I visited the page, it wasn't showing a button. Came back here, clicked again, and it did show a button. Clicking it worked well. I don't know why it's breaking so easily.
  - pasta 10y ago
    ha! you are right. A revisit made the button appear.
- dredmorbius 10y ago
  Prefix the URL with "about:reader?url=" I have to do that a lot. Web design isn't the solution. Web design is the problem.
lordnacho 10y ago
You can see the Pacific Ocean on one side, and you can see the Atlantic on another. You know of something called a shovel, machetes, labourers and engineers. And you have a budget. Of course, it must be possible to dig a trench connecting the two oceans. When you start, you find your men get ill from the climate. And perhaps you don't quite know how to build a reliable lock, but you figure the engineers will find out somehow. And meanwhile there are constraints coming in from government. Things take long because it's easy to see the big picture, and getting more detail (ie learning) normally means discovering that you need a bit more time to do fix some issue that you didn't see standing on the hill.
- jacquesm 10y ago
  Except that in software 1000's of groups have already dug 1000's of canals and yet the next group that will build a canal will insist on doing it their way. If our profession would only be able to simply collect lessons learned and to pass those on to the next generation then we'd be moving faster on average and we'd produce better software. we don't need 500 crappy, slightly different ways of doing the same thing, we need maybe 5, battle tested and hardened bullet proof solutions that are well engineered. Fragmentation is going to be open source's death at some point if this continues, that's one area where closed source actually has an edge: focus. In part the problem is that software producers have managed to convince the buyers of their goods and services that they are never ever liable for their product (something no other industry has managed to do), in part it is because it seems superficially easy to re-invent until you reach the same level of complexity (which inevitably happens) as what those 'archaic' solutions were already addressing. I don't have any answers on how to solve this, unfortunately, but there are software eco-systems that really get this right and others that seem to be totally lost in ever expanding cycles of endless rewrites.
  - meira 10y ago
    A man that never digged a canal don't really master it until he digs some. Software craftmanship isn't something you learn by osmosis.
amelius 10y ago
What is the best resource which explains an outsider why developing software is hard and takes a lot of time in general? I'm thinking of a small accessible book (perhaps even a cartoon) that starts with a bunch of analogies, and then explains why those analogies are true in real life (this last part is important).
- Scarblac 10y ago
  Maybe this? http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/?src=longreads http://www.bloomberg.com/graphics/2015-paul-ford-what-is-cod... But I don't think it catches just how much of an expert on all sorts of quirks of how many libraries a programmer must actually be, like the current article does.
- arethuza 10y ago
  What about "Big Ball of Mud" http://www.laputan.org/mud/ http://www.laputan.org/mud/ Some of the pictures are brilliant.
- vram22 10y ago
  This may not be the "best" (not sure if there is a single best), but is pretty well known: https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month It also has a 20 or 25 year later anniversary edition. I like his chapter / point about "No silver bullet" in the 2nd edition. Edit: He (Fred Brooks, the author) was in charge of the OS/360 project (OS for the IBM System 360 mainframe series). A pretty large project of the time - years long. The cover image is a good analogy - tar pits.
- douche 10y ago
  Perhaps a bit simpler, but all too applicable in real-life. http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-a-programmer/ http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-...
forgottenacc57 10y ago
My most recent project has taken more than 18 months. I implemented 2 major subsystems that I later realized I could simplify the system by throwing out. My product is very technical so there was probably about four months trying to make things work and learning what did and did not work. I had to learn about 75% of the development tools and languages I was building with. I don't like the idea of releasing half baked, half working software that represents half of my vision. I had to fit the project in between my money paying job, family, friends, life. I had lots of great ideas and the fun bit about developing software is when you get to implement those "wow it would be cool if...." functions. I just want to fix bugs after the software is released, not just be starting on the code. I want the software to be so far along that it would cause a competitor to think twice about copying. I think software should be great, not just barely functional, so I added lots of things to polish and make it awesome. Near the end of the build I realized I could make a major leap forward so I redesigned a key area of the UI. I built as much of it as I could before release because after release its MUCH harder to add new features. There you go, that's why it took me so long.
- jacquesm 10y ago
  Kudos on taking the risk to release a bit later and delivering something that meets your higher standards, that's going against accepted business wisdom but it is only because 'everybody else releases barely functional crap too' that that became the norm for competition.
  - forgottenacc57 10y ago
    I get a vision in my head of what the software should be and I'm compelled as though by force to make it match the vision. I don't seem to have much choice.
  - samcal 10y ago
    That's not usually the reason to ship early. Common wisdom is to ship early because it's very possible that you do not have a good understanding of the problem you are trying to solve, or how your users will use your solution. The only way to find a better understanding is to release your product and see how it does. It is usually efficient to find this information out before you spend time polishing the wrong parts of the solution.
hex13 10y ago
It looks for me that despite what author wrote, he WAS indeed in exploratory phase: `So now we have a sequencer device, how do we get events from it? Can we do it in the main loop? Turns out it probably doesn't integrate too well with Qt, (...)` `My initial thought was making a grid of spinners,(...) but then I realized that there isn't an easy way to make headlines in Qt's grid. (...) So after some searching, I found out that it would be better to have a tree view(...)` And many similar things. Exploration and technical spikes - nothing wrong with that but author wrote: ` It's pretty common to do so if you're in an exploratory phase, but in this case, I had a pretty good idea of what I wanted to do right from the start, and that plan seemed to work. ` and then he contradicted himself writing about various difficulties and explorations... (I don't see anything wrong with that, absolutely. I think that exploratory phase is essential to good design).
- icebraining 10y ago
  I think the author means exploratory as in writing throw-away prototypes to test an idea. There's obviously always exploratory work during development, unless you're reading everything from an extremely detailed spec.
koolba 10y ago
Software development is 5% inspiration, 20% perspiration, and 75% procrastination.
- eludwig 10y ago
  >> and 75% procrastination. I laughed at this and it's true. But I would like to champion a bit of procrastination in creative pursuits. I know the way my mind works and the simple truth is that sometimes it pays to wait until my head has worked through what feels like an incessant cycle of building up possible approaches and relentlessly tearing them down. So lets say I'm given an assignment (or a stand-along idea occurs to me). I immediately start to visualize possible outcomes and the approaches that I would need to take to get there. I throw out 99% of these subconsciously. Some of these ideas don't even make it into a full-blown thought, but they are there lurking below the surface. All of this takes some time. Subjectively, it just doesn't feel like the right time to start yet. This may look like procrastination, but it isn't. I'm not arguing for always waiting until every detail is mapped out in my mind to begin an effort, but sometimes it pays to let your head work through stuff for a bit. Of course, moderation in everything (including moderation).
  - MyNameIsFred 10y ago
    Yes! Sometimes I will have a have a big problem that I will decide not to fix immediately, even if it's more "important" than other things I'm working on. Then, when I come back and rapidly write a simple solution, the question of course is "Why didn't you do this a week ago if the solution was so simple?". The answer is that the answer wasn't simple at first. Understandings of a problem and solution often live and die many times over in a mental background thread, before they ever reach a keyboard. Sometimes, when under pressure, I am forced to ignore this fact and start writing the moment I reached the "Yeah, I think that would probably work, let's git 'er done!" phase. In almost every such case, it takes longer overall as I iterate through incomplete solutions and rewrites.
  - majewsky 10y ago
    One of the best parts about my job is that my boss understands that this is how good developers work.
Tistel 10y ago
Its impossible to accurately estimate an unknown. Once you have experience, a rough sketch of the solution will pop into your head right away, but as you go along all these devilish little details will pop up. Also, meetings run by people who can't write code who just want to talk to people for an hour because it breaks up their dull day will eat a decent chunk of time. When starting a new project and in the "give me estimates phase", I do my best, but, I know they are essentially made up numbers. If I were in charge, I would eliminate time estimates. Just have high level goals, keep breaking them down (or up, in bottom up dev) and have lots of automated test cases. I guess like that continuous dev (which I have never done, just read about) style.
- douche 10y ago
  > Also, meetings run by people who can't write code who just want to talk to people for an hour because it breaks up their dull day will eat a decent chunk of time Is there any other reason for weekly status meetings?
CrLf 10y ago
Because software development is actually research disguised as engineering. (And that's why "engineering" when related to software is filled with mumbo-jumbo and cargo-cultism.)
- jacquesm 10y ago
  That's true in some cases, and in those cases time/budget overruns are probably more acceptable than in CRUD app #34.
  - CrLf 10y ago
    It's also true for CRUD app #34, because the hard part is never the implementation, it's dealing with shifting requirements, either due to the nature of businesses, unsufficient understanding of the problem being solved or sabotage from project stakeholders (conflicting agendas, politics, etc.). When you plan to build a bridge between A and B, everybody can see that the problem being solved is connecting A to B. You then have to match material knowledge and construction experience from other bridges with the specifics of the new location. Also, you define project completion when you can cross the bridge. When you start out building a "simple form" to solve some business need, nobody agrees on what that need is or what the form should actually solve. Navigating this is research. Also, nobody can tell when the project is indeed finished (i.e. solves the problem).
    - jacquesm 10y ago
      That does not begin to explain the sometimes orders of magnitude difference between estimates and final costs, elapsed time and required manpower. I've sat in on meetings with people discussing a new project, it is absolutely incredible how much of the future derailing of a project you can see happen in real time during those preliminary meetings. A probable clue you can find in this nice video of a product meeting between 'engineering' and 'customers': https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be
    - bemmu 10y ago
tyingq 10y ago
"Why does software development take so long?" I can see this as a valid question for a subset of problems. For example, the basic database CRUD applications that get built over and over. The closest solutions I've seen were DabbleDB (acquired by Twitter and shuttered) and Quickbase (laudable, but the pricing model doesn't work for many).
0xmohit 10y ago
I'd be tempted to say that a lot depends on being clear on what needs to be achieved. I've personally seen deadlines slip mostly for the lack of clarity on what needs to be done. An analogy might be to say that what was initially described as an elephant turned out to be a giraffe. Of course, there are instances when the actual approach, implementation or choice of technologies tend to weigh. But the latter is much less frequent.
MyNameIsFred 10y ago
> In a sense, programming is all about what your program should do in the first place. The “how” question is just the “what”, moved down the chain of abstractions until it ends up where a computer can understand it, and at that point, the three words “multichannel audio support” have become those 9,000 lines that describe in perfect detail what's going on. This closing paragraph is excellent. While I do wonder if the author's progress along these wouldn't have been accelerated if more of this evolution took place before the first line of code, the overall message here is very true and very well put. When I made this realization for myself, it was a major turning point in the quality, speed, maintainability, and usability of the code I produce, especially when I can successfully define the final code structure to directly reflect this progressive refinement from intent to implementation.
- tremon 10y ago
  I do wonder if the author's progress along these wouldn't have been accelerated if more of this evolution took place before the first line of code Doesn't that require much more thorough interface specifications and documentation than we have now (industry-wide)? I don't do that much software development anymore, but I have always been put off by the amount of trial-and-error required because of ill-defined interfaces.
  - MyNameIsFred 10y ago
    This is true. I must admit that my more recent, positive experiences are only possible due to my accumulated understanding of my dependency chain and toolkit. I work on multiple things, but my largest, indefinitely-running one has only a single dependency...but it's a truly beastly, poorly documented enterprise system. It took me almost 2 years before I properly understood the system's intended usage, interfaces, intents, bugs, and pitfalls. The JavaDocs are awesome in their uselessness: /** * Sets the Wimbulator. * * @param wimbulator * the Wimbulator to set * @param identifier * the identifier to use * @param flags * flags * @param legacy * used to apply legacy */ public void setWimbulator(Wimbulator wimbulator, String identifier, long flags, boolean legacy);
mloranger2 10y ago
Okay, so developing a "multi channel audio support" feature took three months. Is that expensive? What would this type of feature cost to implement without software? Answer: a LOT or even borderline impossible. Look, to a large degree software like this replaces what used to have to be done by electronics (or, more likely, not at all). The fact that it took three months is a borderline miracle. Oh, you want to make this process faster? Okay -- time to get the AI community involved because I don't really see how human beings are going to perform any faster than they already are. Look, my point is that anyone who claims that this could be done faster or is the result of "bad engineers" or anything else is basically delusional IMHO. 'Nuff said.
draw_down 10y ago
Because nobody really knows what they are doing.
- jacquesm 10y ago
  That's roughly the gist of it. There are exceptions but not many.
dqdo 10y ago
Software development takes a long time because of the users. If you think about every piece of software that has ever been written, they have been created to solve a human problem or resolve a human concern. As we build more impressive systems, our expectations also change. A state of the art website just 5 years ago is unacceptable today. Think about all the work that we now have to do in order to make things mobile responsive. Features such as chat and video, which were revolutionary in 2004-2005 (when Youtube and Facebook were found) are common requirements of projects today. In additional to an increase in user expectations, we have to understand that when we undertake a project it is to solve a new problem whose implications cannot fully be understood from the beginning. In most software projects, the problems only becomes clear after spending months building it. Once we have the software as an artifact, we will find new ideas and new extensions that we could have never imagined before. Additionally, a large part of our software has to interact with other artifacts in the real world. These artifacts are built by other people and continuously change. A large part of maintaining a project is ensuring that is still compatible with all the programs that it depends on.
- icebraining 10y ago
  Features such as chat and video, which were revolutionary in 2004-2005 They were revolutionary in 1968: https://www.youtube.com/watch?v=yJDv-zdhzMY https://www.youtube.com/watch?v=yJDv-zdhzMY
  - wolfgke 10y ago
    Chat and video was revolutionary in 2004-2005 on the web.
    - jacquesm 10y ago
      The hell they were, I pioneered that in 1995.
wickedlogic 10y ago
Software development takes a long time because we write code still. Thats essentially the crux of it. Humans then plan/optimize efforts on the wrong layers, and underestimate the cost of updating code over time. Surprisingly so at scale. The systems that tend to produce the fastest (that also last) results are the one's that give the developer the most control over the environment to get the task done (LISPs). Or one's that generate code for you, and let you deal with abstract flows as the control unit. We don't really use either of those things in industry... but niche segments do to great success.
Edmond 10y ago
I think a big part of the problem can be attributed to tool makers and the lack of true innovation in that arena. As a developer tool maker (HiveMind: crudzilla.com) I am almost always disappointed when I come across a "new" development tool and find that it is the same tired code editor tricks (key bindings, syntax highlighting, code completion, multi-pane editing) with some "flavor of the day" gimmick (git) thrown in. IDEs have been around for more than 20 yrs and the most popular IDEs have not advanced the state of software development much at all. Compare IDEs with chip fabrication as an easy to compare example. Improvements and advances in chips can be directly tied to advances in fabrication, the same can't be said for software because there is almost no innovation happening in those products despite appearances.
- jbattle 10y ago
  I'd love to see some advances in IDEs, but the author is saying pretty clearly that the bulk of the time taken was in high-level decision making, not in struggling to get his ideas into the computer. > "In a sense, programming is all about what your program should do in the first place."
  - tonyedgecombe 10y ago
    Yes, good tooling can shave minutes or hours off here and there, maybe pick out a few bugs but it won't save you from spending months or years writing an ill-conceived product.
  - Edmond 10y ago
    That statement in a way makes my point. Good tooling should help the creator offload a lot of the "thinking about your program", rather it should help you focus on what you want to do. Also the long time taken in high-level decision making is strongly tied to the difficulty of getting even a trivial piece of software working, even if it isn't always apparent. For instance if the output of the high-level decision making was to create a spreadsheet or PowerPoint presentation, I am sure it wouldn't take anywhere as long. I have a few long rants on this topic that I'll be putting on the Crudzilla blog (blog.crudzilla.com), stay tuned :)
  - 10y ago
tsewlliw 10y ago
Why does some software development take so little time? My suggestion for the answer is that when the goal of the software project is selected carefully in the context of the surrounding ecosystem you can connect a few things and all the real magic was in picking the goal. When instead you pick the goal and then figure out how to achieve it, who would be surprised that there is often more work involved after picking the goal?
seanwilson 10y ago
If you were delivering very similar projects several times (in which case you should be reusing existing code anyway), estimates would be a lot easier and more accurate. As most software projects are using a combination of tools, people, requirements etc. that have never been combined before, accurate estimates are always going to be difficult. I routinely read that estimates get better with experience. The only way I see this as true is in that you learn to broaden your estimates the more risk that is involved, not that you eventually are able to give accurate and narrow estimates.
lazyjones 10y ago
It doesn't really, unless you overgeneralize your solution (just like the rhetorical question in the title) and make wrong toolset choices. 20 years ago we also had fewer choices, hence fewer bug-ridden abstraction layers and moving targets. We mastered one programming environment and target platform, both were much less complex than what we have today. We could also tack something together quickly in an awkward way without putting the code on a world-visible platform as part of our resume. Also, more precautions are necessary today, like security, general code quality (no more tricks that work only on one compiler).
p0nce 10y ago
Business-time goes seemingly faster than engineering-time. Engineering requires luxurious calm and ample amounts of time to achieve anything of value. Hence the disconnection, the business will ask "why do engineers take so much time?" while the engineers ask themselves "why does the management changes the plan so much?". But a key point to have successful engineering is not panicking upon this perceived, unbearable, uncontrollable slowness and still invest in quality: capital instead of debt.
shados 10y ago
If you read the comments so far, everyone has their favorite. The answer is probably "all of the above". That said, my personal favorites: Most software engineers are bad and refuse to admit it to themselves (which would let them get better). Too much ego. Given a large enough department/team, only a small percentage is actually doing significant work. Yes, even at Google/Facebook/Amazon/whatever. Second, software engineers are extremely conservative. For all the flack fast moving ecosystems like JavaScript get, the only reason we see so many iterations is that the end goal is visible from far, but people reject it. So we just make 100 intermediate steps to get there. Eg: a subset of functional programming paradigms and type systems. Think TypeScript...which has to be comfortable for the peanut gallery, while trying to support advanced features many know need to be there in a modern language, and the struggle between those 2 goals. That ensures we're going slower than we need to, while basically ensuring those techs will be obsolete sooner than later (because we need a stepping stone to the "right" solution as to not alienate developers)
- gmarx 10y ago
  Came here to make your second point. Most developers are bad and think they are good. I'll add that most of the people who provide requirements don't understand software development and most developers refuse or are unable to understand the domain space and yet insist that the requirements writers just tell them the "what" and leave the "how" to the developers. Oh- and using a process name, such as "agile" as an excuse rather than a constraint.
- clifanatic 10y ago
  > Most software engineers are bad and refuse to admit it to themselves (which would let them get better) Well… not to be confrontational, but in the context of this post, you seem to be implying that if most software engineers (which would, statistically speaking, include me) were “better”, they could put software together faster. I have to say, the implication bothers me, as does my suspicion that it’s a sentiment shared by most non-programmers out there. The problem is, there is a lower limit on how long it can possibly take to complete a programming task - and I think even the MBAs will concede that that lower limit is higher than the time spent typing out the words that make up the program. So, if we accept the premise that “good programmer” = “fast programmer” with no reasonable, objective measure of how fast fast is, the people doing the judging will always end up using the metric “how long I wanted it to take”, which is usually about a day for any task.
jbb555 10y ago
People STILL confuse the construction of software with the construction of buildings. We can estimate fairly accurately how long it will take to build a building once we have reasonable plans for it. I can pretty accurately say that it will take about 4 minutes to build the software once I have the plans to build it. The compiler pretty much automates the whole job. Writing software is NOT construction. Much of it isn't even design. Most of it is gathering detailed requirements and writing them down in unambiguous form (code). My asking how long it's going to take to write a software it's like saying to a building contractor how long will it take to design every single detail of a city block including gathering all the requirements. Also the requirements for software are much more detailed than building. 100000 lines of code represents 100000 decisions. I bet not many buildings have 100000 decisions. And 10000 is tiny for a software project.
- yannickt 10y ago
  There is a great essay by Jack Reeves making a similar point. What Is Software Design? http://www.developerdotstar.com/mag/articles/reeves_design.html http://www.developerdotstar.com/mag/articles/reeves_design.h... FTA: There is one consequence of considering code as software design that completely overwhelms all others. It is so important and so obvious that it is a total blind spot for most software organizations. This is the fact that software is cheap to build. It does not qualify as inexpensive; it is so cheap it is almost free. If source code is a software design, then actually building software is done by compilers and linkers.
  - someguydave 10y ago
    But then you could argue that the machine code emitted from the compiler is a design and the actual hardware that implements it is "building the software".
    - someguydave 10y ago
      But then you could argue that the implementing transistor arrangement is a design and the actual movement of electrons that implements it is "building the software"
Futurebot 10y ago
- Requirements that change mid-stream, dozens, sometimes hundreds of times - Team has to do speculative development for ill-formed or understood problem domain - Requirements that are improperly gathered - Requirements that are underspecified - Poorly understood problem in general - Fighting bugs in system libraries - Fighting bugs in third party libraries - Fighting issues with integration on deployment and/or third party anything - Complex underlying technology requiring its own kind of discovery (like in the OP) - Poor/incorrect documentation - Project members that go off on their own, refuse to communicate, deviate wildly from the project style, break things - Management that insists on doing things from scratch - Management that insists on using buggy, broken, etc. 3rd-party or even in-house systems/libraries for political or faux-business reasons - Constant software developer interruptions - Long meetings that consist of circular discussions, politics, back-and-forth arguments about personal preferences disguised as "important project stuff" - Administrative minutiae that never stops adding up - People leave a project partway through, taking project knowledge with them - Broken/buggy tools/environments/systems - People getting pulled off a project to work on another project - Bad use of project management, source code management, other tools - Insistence on wheel reinvention for well-understood problems for career advancement, resume building, or boredom-relief purposes - Low morale / poor focus due to bad management, co-workers, bad tools, bad work environment - Low morale / poor focus due to reliance on: late night work (leading to insufficient sleep), weekend work, little/no time off or breaks, reliance on crunch/death marches - Dealing with company/project politics in general - Lack of focus due to team member resume-polishing/interviewing part way through a bad project - Occasional project sabotage
- majewsky 10y ago
  Hello, Mr. Brooks. Didn't see you coming in.
ausjke 10y ago
Maybe most programmers are amateurs? If you're really good at data-structure/algorithms/etc and practice them like a surgeon frequently, maybe the landscape will be very different. "Anybody can code, even an idiot can learn to code in 24 hours"
njharman 10y ago
Cause I (we?) spend too much time reading articles like this instead of coding.
Pica_soO 10y ago
Because a lot of stuff gets written several times. Doesent matter wether its agile or waterfall, and i dont care who started it - customers or software-engineering, wether the rewrite happened before roll out, or while the thing was already in the wild. Software development takes so long, because we tear down what we build and redo it, alot of times over.
deleted 10y ago
[deleted]
jimjimjim 10y ago
Following on the analogies already presented here: Coding is not construction (that's the compiler). Coding is the architect/drafter. If you want a pre-planned house, no problem, that's the same as buying shrink wrap. If you want to add a swimming pool under then that a whole bunch of conversations, customization and checking. If you want a completely bespoke house then then that is a conversation that lasts for months and the requirements will always change. Think about that Grand Designs tv show. that is software and the architect is software development
imichael 10y ago
To me it all boils down to this: hardware is all about making billions of things, all identical. Software is about building innumerable things on top of that hardware, all different. What would you like it to do, what can be done, how to do it, how to organize a group of people to do it, how to make sure it works and stays working, it's all creative work and it takes time to do it well.
drinchev 10y ago
Once I outsourced my side-project, since the pay per hour ratio worked better for me, compared to what I earn, combined with the complexity of that project. The "only" thing I had to do is write specs. The specs took me 3 weeks to write ( finding a lot of logical "bugs" in my project ), but the detail I went to ( SQL schema suggestions, framework suggestions, front-end plugins suggestions, ready pixel perfect design with different states, urls for the different pages, etc. ) was one of the biggest benefit for the final price. The company that took the project estimated 4 months and delivered it in 3 and half. They said those were the best specs they've ever seen. Moral of the story : Do full specs before you start coding ( I know it's boring ).
- VladimirGolovin 10y ago
  I did the same for my outsourced side-project - I went full waterfall. The spec turned out to be 250 pages long (without pics) and took almost a year to fully develop from the conception stage. In addition to this, I hired a designer to develop an additional design spec for the UI. So far, this worked great. My total time spent on this project since the moment I gave the spec to developers is about one workday in total. The questions from the devs are mostly trivial, and solvable in an hour or less.
keithnz 10y ago
my real takeaway from this article is the amazing feat of typing at 800 characters a minute! according to http://smallbusiness.chron.com/good-typing-speed-per-minute-71789.html http://smallbusiness.chron.com/good-typing-speed-per-minute-... a good typist does around 335 cpm so this is way over twice as fast is this a typo and supposed to be 80? that would be very slow.
- Nadya 10y ago
  My guess is that is with "bursting" and is not very sustainable. Not to mention, being able to type so quickly is very impractical when you need to think about what to say (well....type) and with programming especially, typing speed is irrelevant. To give a comparison - I can type 700~ cpm steadily, which means I can maintain it for large periods of time. Only practical when copying text when OCR fails me or when doing data entry. With "bursts" (unsustainable but rapid typing) I can type 160-170wpm with a negligible error rate (<1%) but I can only keep it up for a few minutes. Realistically, I type closer to 110-120wpm, because I take longer pauses between words while I consider what I am about to say (again....type). My hunch is the author actually meant 800cpm, but I do have my doubts to the accuracy or susustainability of the figure. It isn't "impossible" but it is "The top 1% of the top 1%" levels. :) http://i.imgur.com/NW2Kq4p.png http://i.imgur.com/NW2Kq4p.png
oneplane 10y ago
Because software is a half-visible moving target, that's why.
- prefect42 10y ago
  I like your answer the best, quite succinct. Software is / has way too many intangibles, by its nature. If you try to show the customer some tangible progress every three weeks, that's all very good, but strongly highlights the "intangiblesness" (sorry made up word) of it all. Once the customer starts seeing some "real" feature take shape, then the feature requests really start pouring in, and any previous understanding of this feature before this point, becomes a dim ephemeral dream in everyone's minds. Constant moving target.
deleted 10y ago
[deleted]
stevefeinstein 10y ago
It doesn't. It takes as long as it takes. It's a constantly evolving moving target. Why does writing a book or making a movie take so long? It's a creative thing. It's hard.
dockd 10y ago
I suspect it has something to do with a) The programmer not actually using the software they create. They end up spending time trying to understand the domain. b) The programmer uses the software they create and wants it to do everything (see also Second System Effect, Zawinski's Law/Law of Software Envelopment). c) The programmer expects components to work as documented.
deleted 10y ago
[deleted]
eagsalazar2 10y ago
Buildings are totally different in that they are all, roughly, the same or small variations on the same design. Consequently it is very easy to reuse massive amounts of reusable components in extremely predictable ways and to define standards for review that can be broadly applied across the majority of projects. This is not the case when builders are doing something that is actually totally novel in that they don't have many existing examples and it is built of entirely custom components. In cases like that it is actually exactly like software in that there is a lot of figuring things out as you go along, schedules are usually laughably off by orders of magnitude, and there are frequently major screw ups (Big dig, Seattle big bertha fiasco, SF Bay Bridge, ... to name a very few) Of course in software we aren't always reinventing everything as people unfairly complain. We reuse way more than we reinvent - postgres, elixir, phoenix, browsers, OSX, etc, etc, etc represent 1000s of lifetimes of work and the majority of the totality of work behind an actual product. The time we do spend, and consequently the variability in schedule, is due to the fact that most software is actually quite different in its requirements from all previous software and usually previous examples (aka competitors) have not made source code available for reuse. One huge difference between software development and "custom" building development is that good software developers embrace this variability and uncertainty - hence agile. Most other fields of engineering are still caught in an archaic and delusional world of gantt charts and hard deadlines with feature lists developed by architects. It never works that way insofar as the work is _custom_ - in any field! The thing I really hate about this conversation is how so many people are so eager to just chalk all this up to software developers being half-assed hacks. Well there are lots of teams at places like HP staffed by only "certified" software engineers and guess what? They still are terrible at predicting outcomes and they are actually far far worst in general at their jobs than small, modern, agile teams who make the best of the chaos by addressing that reality in how they work (again, agile). This is software development people. It isn't about bad devs and lack of certification, it is just hard. For comparison I did integrated circuit design for 7 years right out of school up through being a lead on multiple larger teams before switching to software. The rigor you perceive in other areas of engineering is a total fantasy. Bridges and buildings are maybe the exception but again this is only possible because it is such a canned art by comparison. If you disagree please just enumerate some specific "building codes" or "certification requirements" that you think would actually make any difference on your average complex software project.
ilaksh 10y ago
Why does it need another thread if ALSA uses poll()?

296 points by cheiVia0 10y ago

scrrr 10y ago

Because it's an art. It's like painting a picture. Good art takes time.

forgottenacc57 10y ago
This is in fact the correct answer.
jacquesm 10y ago
You're just asking for it: http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm http://www.idlewords.com/2005/04/dabblers_and_blowhards.htm
- tempodox 10y ago
  One (half-) sentence from that article especially caught my eye: ...computer programmers create artifacts that have to stand up to an objective reality. If we could make this more of a practise than a theory, it might substantially improve the state of things.
- rkeene2 10y ago
  Reading through the article, and having no way to comment there, I'll comment here: > It is true that both painters and programmers make things, just like a pastry chef makes a wedding cake, or a chicken makes an egg. But nothing about what they make, the purposes it serves, or how they go about doing it is in any way similar. This misses the mark almost entirely, from my understanding. Hackers and painters CREATE something -- it does not exist and then it does. The wording Paul Graham uses is "makers", but it's not what I feel so maybe that is the criticism being expressed in this part of the article ? The purpose of artists who paint and hackers /IS/ the same -- to create. To get what is inside of you out. To have the expression which has no form to take form and be in the world as something that can be shared with another soul. The methods are the same -- extract what is inside of you into concrete form, mold it, remove the bits that are not right, add more bits, move the bits around. Certainly this is not true of computer programmers, which the article refers to hackers as but this is a failure of the article not of the original comparison. Nor is it true of all painters, since the word is too generic, which is a failure of Paul Graham's choice of wording for this comparison since based on the rest of the reading that is what he is referring to. Computer programmers program computers, perhaps soullessly. Painters paint things, perhaps soullessly. Hackers hack because they need to. Artists that paint paint because they need to. As to the rest, I don't know if Paul Graham's intention was to attempt to borrow "coolness" from artists, but for me it is not. It's an attempt to explain why one must hack and why that is an identity and not an activity or a profession.

rogual 10y ago

And this is just for a sole developer who knows what he's doing. When you have a team of people at varying levels of ability, a meeting for every decision, JIRA-wielding project managers with burndown charts to keep happy, standards & practices, legal, translators, UI designers, all regularly taking out exclusive locks on each other's time... it's a wonder anything gets done at all.

hex13 10y ago
It's correct. I've noticed that when I work in a team I work with maybe 10% of my solo efficiency. Just because of this whole team work bullshit and because of legacy code (it takes me often more time to understand how code written by someone works than it would take to write such code from scratch).

pasta 10y ago

The 'readable' button in Firefox was not working for me. Maybe others also would like to paste this in the url bar: javascript:document.body.style.width='600px';document.body.style.margin='0 auto';void(0);

StavrosK 10y ago
Firefox is wonky with its "readable view" button, huh? I visited the page, it wasn't showing a button. Came back here, clicked again, and it did show a button. Clicking it worked well. I don't know why it's breaking so easily.
- pasta 10y ago
  ha! you are right. A revisit made the button appear.
dredmorbius 10y ago
Prefix the URL with "about:reader?url=" I have to do that a lot. Web design isn't the solution. Web design is the problem.

lordnacho 10y ago

You can see the Pacific Ocean on one side, and you can see the Atlantic on another. You know of something called a shovel, machetes, labourers and engineers. And you have a budget. Of course, it must be possible to dig a trench connecting the two oceans. When you start, you find your men get ill from the climate. And perhaps you don't quite know how to build a reliable lock, but you figure the engineers will find out somehow. And meanwhile there are constraints coming in from government. Things take long because it's easy to see the big picture, and getting more detail (ie learning) normally means discovering that you need a bit more time to do fix some issue that you didn't see standing on the hill.

jacquesm 10y ago
Except that in software 1000's of groups have already dug 1000's of canals and yet the next group that will build a canal will insist on doing it their way. If our profession would only be able to simply collect lessons learned and to pass those on to the next generation then we'd be moving faster on average and we'd produce better software. we don't need 500 crappy, slightly different ways of doing the same thing, we need maybe 5, battle tested and hardened bullet proof solutions that are well engineered. Fragmentation is going to be open source's death at some point if this continues, that's one area where closed source actually has an edge: focus. In part the problem is that software producers have managed to convince the buyers of their goods and services that they are never ever liable for their product (something no other industry has managed to do), in part it is because it seems superficially easy to re-invent until you reach the same level of complexity (which inevitably happens) as what those 'archaic' solutions were already addressing. I don't have any answers on how to solve this, unfortunately, but there are software eco-systems that really get this right and others that seem to be totally lost in ever expanding cycles of endless rewrites.
- meira 10y ago
  A man that never digged a canal don't really master it until he digs some. Software craftmanship isn't something you learn by osmosis.

amelius 10y ago

What is the best resource which explains an outsider why developing software is hard and takes a lot of time in general? I'm thinking of a small accessible book (perhaps even a cartoon) that starts with a bunch of analogies, and then explains why those analogies are true in real life (this last part is important).

Scarblac 10y ago
Maybe this? http://www.bloomberg.com/graphics/2015-paul-ford-what-is-code/?src=longreads http://www.bloomberg.com/graphics/2015-paul-ford-what-is-cod... But I don't think it catches just how much of an expert on all sorts of quirks of how many libraries a programmer must actually be, like the current article does.
arethuza 10y ago
What about "Big Ball of Mud" http://www.laputan.org/mud/ http://www.laputan.org/mud/ Some of the pictures are brilliant.
vram22 10y ago
This may not be the "best" (not sure if there is a single best), but is pretty well known: https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month https://en.m.wikipedia.org/wiki/The_Mythical_Man-Month It also has a 20 or 25 year later anniversary edition. I like his chapter / point about "No silver bullet" in the 2nd edition. Edit: He (Fred Brooks, the author) was in charge of the OS/360 project (OS for the IBM System 360 mainframe series). A pretty large project of the time - years long. The cover image is a good analogy - tar pits.
douche 10y ago
Perhaps a bit simpler, but all too applicable in real-life. http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-a-programmer/ http://heeris.id.au/2013/this-is-why-you-shouldnt-interrupt-...

forgottenacc57 10y ago

My most recent project has taken more than 18 months. I implemented 2 major subsystems that I later realized I could simplify the system by throwing out. My product is very technical so there was probably about four months trying to make things work and learning what did and did not work. I had to learn about 75% of the development tools and languages I was building with. I don't like the idea of releasing half baked, half working software that represents half of my vision. I had to fit the project in between my money paying job, family, friends, life. I had lots of great ideas and the fun bit about developing software is when you get to implement those "wow it would be cool if...." functions. I just want to fix bugs after the software is released, not just be starting on the code. I want the software to be so far along that it would cause a competitor to think twice about copying. I think software should be great, not just barely functional, so I added lots of things to polish and make it awesome. Near the end of the build I realized I could make a major leap forward so I redesigned a key area of the UI. I built as much of it as I could before release because after release its MUCH harder to add new features. There you go, that's why it took me so long.

jacquesm 10y ago
Kudos on taking the risk to release a bit later and delivering something that meets your higher standards, that's going against accepted business wisdom but it is only because 'everybody else releases barely functional crap too' that that became the norm for competition.
- forgottenacc57 10y ago
  I get a vision in my head of what the software should be and I'm compelled as though by force to make it match the vision. I don't seem to have much choice.
- samcal 10y ago
  That's not usually the reason to ship early. Common wisdom is to ship early because it's very possible that you do not have a good understanding of the problem you are trying to solve, or how your users will use your solution. The only way to find a better understanding is to release your product and see how it does. It is usually efficient to find this information out before you spend time polishing the wrong parts of the solution.

hex13 10y ago

It looks for me that despite what author wrote, he WAS indeed in exploratory phase: `So now we have a sequencer device, how do we get events from it? Can we do it in the main loop? Turns out it probably doesn't integrate too well with Qt, (...)` `My initial thought was making a grid of spinners,(...) but then I realized that there isn't an easy way to make headlines in Qt's grid. (...) So after some searching, I found out that it would be better to have a tree view(...)` And many similar things. Exploration and technical spikes - nothing wrong with that but author wrote: ` It's pretty common to do so if you're in an exploratory phase, but in this case, I had a pretty good idea of what I wanted to do right from the start, and that plan seemed to work. ` and then he contradicted himself writing about various difficulties and explorations... (I don't see anything wrong with that, absolutely. I think that exploratory phase is essential to good design).

icebraining 10y ago
I think the author means exploratory as in writing throw-away prototypes to test an idea. There's obviously always exploratory work during development, unless you're reading everything from an extremely detailed spec.

koolba 10y ago

Software development is 5% inspiration, 20% perspiration, and 75% procrastination.

eludwig 10y ago
>> and 75% procrastination. I laughed at this and it's true. But I would like to champion a bit of procrastination in creative pursuits. I know the way my mind works and the simple truth is that sometimes it pays to wait until my head has worked through what feels like an incessant cycle of building up possible approaches and relentlessly tearing them down. So lets say I'm given an assignment (or a stand-along idea occurs to me). I immediately start to visualize possible outcomes and the approaches that I would need to take to get there. I throw out 99% of these subconsciously. Some of these ideas don't even make it into a full-blown thought, but they are there lurking below the surface. All of this takes some time. Subjectively, it just doesn't feel like the right time to start yet. This may look like procrastination, but it isn't. I'm not arguing for always waiting until every detail is mapped out in my mind to begin an effort, but sometimes it pays to let your head work through stuff for a bit. Of course, moderation in everything (including moderation).
- MyNameIsFred 10y ago
  Yes! Sometimes I will have a have a big problem that I will decide not to fix immediately, even if it's more "important" than other things I'm working on. Then, when I come back and rapidly write a simple solution, the question of course is "Why didn't you do this a week ago if the solution was so simple?". The answer is that the answer wasn't simple at first. Understandings of a problem and solution often live and die many times over in a mental background thread, before they ever reach a keyboard. Sometimes, when under pressure, I am forced to ignore this fact and start writing the moment I reached the "Yeah, I think that would probably work, let's git 'er done!" phase. In almost every such case, it takes longer overall as I iterate through incomplete solutions and rewrites.
- majewsky 10y ago
  One of the best parts about my job is that my boss understands that this is how good developers work.

Tistel 10y ago

Its impossible to accurately estimate an unknown. Once you have experience, a rough sketch of the solution will pop into your head right away, but as you go along all these devilish little details will pop up. Also, meetings run by people who can't write code who just want to talk to people for an hour because it breaks up their dull day will eat a decent chunk of time. When starting a new project and in the "give me estimates phase", I do my best, but, I know they are essentially made up numbers. If I were in charge, I would eliminate time estimates. Just have high level goals, keep breaking them down (or up, in bottom up dev) and have lots of automated test cases. I guess like that continuous dev (which I have never done, just read about) style.

douche 10y ago
> Also, meetings run by people who can't write code who just want to talk to people for an hour because it breaks up their dull day will eat a decent chunk of time Is there any other reason for weekly status meetings?

CrLf 10y ago

Because software development is actually research disguised as engineering. (And that's why "engineering" when related to software is filled with mumbo-jumbo and cargo-cultism.)

jacquesm 10y ago
That's true in some cases, and in those cases time/budget overruns are probably more acceptable than in CRUD app #34.
- CrLf 10y ago
  It's also true for CRUD app #34, because the hard part is never the implementation, it's dealing with shifting requirements, either due to the nature of businesses, unsufficient understanding of the problem being solved or sabotage from project stakeholders (conflicting agendas, politics, etc.). When you plan to build a bridge between A and B, everybody can see that the problem being solved is connecting A to B. You then have to match material knowledge and construction experience from other bridges with the specifics of the new location. Also, you define project completion when you can cross the bridge. When you start out building a "simple form" to solve some business need, nobody agrees on what that need is or what the form should actually solve. Navigating this is research. Also, nobody can tell when the project is indeed finished (i.e. solves the problem).
  - jacquesm 10y ago
    That does not begin to explain the sometimes orders of magnitude difference between estimates and final costs, elapsed time and required manpower. I've sat in on meetings with people discussing a new project, it is absolutely incredible how much of the future derailing of a project you can see happen in real time during those preliminary meetings. A probable clue you can find in this nice video of a product meeting between 'engineering' and 'customers': https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be https://www.youtube.com/watch?v=BKorP55Aqvg&feature=youtu.be
  - bemmu 10y ago

tyingq 10y ago

"Why does software development take so long?" I can see this as a valid question for a subset of problems. For example, the basic database CRUD applications that get built over and over. The closest solutions I've seen were DabbleDB (acquired by Twitter and shuttered) and Quickbase (laudable, but the pricing model doesn't work for many).

0xmohit 10y ago

I'd be tempted to say that a lot depends on being clear on what needs to be achieved. I've personally seen deadlines slip mostly for the lack of clarity on what needs to be done. An analogy might be to say that what was initially described as an elephant turned out to be a giraffe. Of course, there are instances when the actual approach, implementation or choice of technologies tend to weigh. But the latter is much less frequent.

MyNameIsFred 10y ago

> In a sense, programming is all about what your program should do in the first place. The “how” question is just the “what”, moved down the chain of abstractions until it ends up where a computer can understand it, and at that point, the three words “multichannel audio support” have become those 9,000 lines that describe in perfect detail what's going on. This closing paragraph is excellent. While I do wonder if the author's progress along these wouldn't have been accelerated if more of this evolution took place before the first line of code, the overall message here is very true and very well put. When I made this realization for myself, it was a major turning point in the quality, speed, maintainability, and usability of the code I produce, especially when I can successfully define the final code structure to directly reflect this progressive refinement from intent to implementation.

tremon 10y ago
I do wonder if the author's progress along these wouldn't have been accelerated if more of this evolution took place before the first line of code Doesn't that require much more thorough interface specifications and documentation than we have now (industry-wide)? I don't do that much software development anymore, but I have always been put off by the amount of trial-and-error required because of ill-defined interfaces.
- MyNameIsFred 10y ago
  This is true. I must admit that my more recent, positive experiences are only possible due to my accumulated understanding of my dependency chain and toolkit. I work on multiple things, but my largest, indefinitely-running one has only a single dependency...but it's a truly beastly, poorly documented enterprise system. It took me almost 2 years before I properly understood the system's intended usage, interfaces, intents, bugs, and pitfalls. The JavaDocs are awesome in their uselessness: /** * Sets the Wimbulator. * * @param wimbulator * the Wimbulator to set * @param identifier * the identifier to use * @param flags * flags * @param legacy * used to apply legacy */ public void setWimbulator(Wimbulator wimbulator, String identifier, long flags, boolean legacy);

mloranger2 10y ago

Okay, so developing a "multi channel audio support" feature took three months. Is that expensive? What would this type of feature cost to implement without software? Answer: a LOT or even borderline impossible. Look, to a large degree software like this replaces what used to have to be done by electronics (or, more likely, not at all). The fact that it took three months is a borderline miracle. Oh, you want to make this process faster? Okay -- time to get the AI community involved because I don't really see how human beings are going to perform any faster than they already are. Look, my point is that anyone who claims that this could be done faster or is the result of "bad engineers" or anything else is basically delusional IMHO. 'Nuff said.

draw_down 10y ago

Because nobody really knows what they are doing.

jacquesm 10y ago
That's roughly the gist of it. There are exceptions but not many.

dqdo 10y ago

Software development takes a long time because of the users. If you think about every piece of software that has ever been written, they have been created to solve a human problem or resolve a human concern. As we build more impressive systems, our expectations also change. A state of the art website just 5 years ago is unacceptable today. Think about all the work that we now have to do in order to make things mobile responsive. Features such as chat and video, which were revolutionary in 2004-2005 (when Youtube and Facebook were found) are common requirements of projects today. In additional to an increase in user expectations, we have to understand that when we undertake a project it is to solve a new problem whose implications cannot fully be understood from the beginning. In most software projects, the problems only becomes clear after spending months building it. Once we have the software as an artifact, we will find new ideas and new extensions that we could have never imagined before. Additionally, a large part of our software has to interact with other artifacts in the real world. These artifacts are built by other people and continuously change. A large part of maintaining a project is ensuring that is still compatible with all the programs that it depends on.

icebraining 10y ago
Features such as chat and video, which were revolutionary in 2004-2005 They were revolutionary in 1968: https://www.youtube.com/watch?v=yJDv-zdhzMY https://www.youtube.com/watch?v=yJDv-zdhzMY
- wolfgke 10y ago
  Chat and video was revolutionary in 2004-2005 on the web.
  - jacquesm 10y ago
    The hell they were, I pioneered that in 1995.

wickedlogic 10y ago

Software development takes a long time because we write code still. Thats essentially the crux of it. Humans then plan/optimize efforts on the wrong layers, and underestimate the cost of updating code over time. Surprisingly so at scale. The systems that tend to produce the fastest (that also last) results are the one's that give the developer the most control over the environment to get the task done (LISPs). Or one's that generate code for you, and let you deal with abstract flows as the control unit. We don't really use either of those things in industry... but niche segments do to great success.

Edmond 10y ago

I think a big part of the problem can be attributed to tool makers and the lack of true innovation in that arena. As a developer tool maker (HiveMind: crudzilla.com) I am almost always disappointed when I come across a "new" development tool and find that it is the same tired code editor tricks (key bindings, syntax highlighting, code completion, multi-pane editing) with some "flavor of the day" gimmick (git) thrown in. IDEs have been around for more than 20 yrs and the most popular IDEs have not advanced the state of software development much at all. Compare IDEs with chip fabrication as an easy to compare example. Improvements and advances in chips can be directly tied to advances in fabrication, the same can't be said for software because there is almost no innovation happening in those products despite appearances.

jbattle 10y ago
I'd love to see some advances in IDEs, but the author is saying pretty clearly that the bulk of the time taken was in high-level decision making, not in struggling to get his ideas into the computer. > "In a sense, programming is all about what your program should do in the first place."
- tonyedgecombe 10y ago
  Yes, good tooling can shave minutes or hours off here and there, maybe pick out a few bugs but it won't save you from spending months or years writing an ill-conceived product.
- Edmond 10y ago
  That statement in a way makes my point. Good tooling should help the creator offload a lot of the "thinking about your program", rather it should help you focus on what you want to do. Also the long time taken in high-level decision making is strongly tied to the difficulty of getting even a trivial piece of software working, even if it isn't always apparent. For instance if the output of the high-level decision making was to create a spreadsheet or PowerPoint presentation, I am sure it wouldn't take anywhere as long. I have a few long rants on this topic that I'll be putting on the Crudzilla blog (blog.crudzilla.com), stay tuned :)
- 10y ago

tsewlliw 10y ago

Why does some software development take so little time? My suggestion for the answer is that when the goal of the software project is selected carefully in the context of the surrounding ecosystem you can connect a few things and all the real magic was in picking the goal. When instead you pick the goal and then figure out how to achieve it, who would be surprised that there is often more work involved after picking the goal?

seanwilson 10y ago

If you were delivering very similar projects several times (in which case you should be reusing existing code anyway), estimates would be a lot easier and more accurate. As most software projects are using a combination of tools, people, requirements etc. that have never been combined before, accurate estimates are always going to be difficult. I routinely read that estimates get better with experience. The only way I see this as true is in that you learn to broaden your estimates the more risk that is involved, not that you eventually are able to give accurate and narrow estimates.

lazyjones 10y ago

It doesn't really, unless you overgeneralize your solution (just like the rhetorical question in the title) and make wrong toolset choices. 20 years ago we also had fewer choices, hence fewer bug-ridden abstraction layers and moving targets. We mastered one programming environment and target platform, both were much less complex than what we have today. We could also tack something together quickly in an awkward way without putting the code on a world-visible platform as part of our resume. Also, more precautions are necessary today, like security, general code quality (no more tricks that work only on one compiler).

p0nce 10y ago

Business-time goes seemingly faster than engineering-time. Engineering requires luxurious calm and ample amounts of time to achieve anything of value. Hence the disconnection, the business will ask "why do engineers take so much time?" while the engineers ask themselves "why does the management changes the plan so much?". But a key point to have successful engineering is not panicking upon this perceived, unbearable, uncontrollable slowness and still invest in quality: capital instead of debt.

shados 10y ago

If you read the comments so far, everyone has their favorite. The answer is probably "all of the above". That said, my personal favorites: Most software engineers are bad and refuse to admit it to themselves (which would let them get better). Too much ego. Given a large enough department/team, only a small percentage is actually doing significant work. Yes, even at Google/Facebook/Amazon/whatever. Second, software engineers are extremely conservative. For all the flack fast moving ecosystems like JavaScript get, the only reason we see so many iterations is that the end goal is visible from far, but people reject it. So we just make 100 intermediate steps to get there. Eg: a subset of functional programming paradigms and type systems. Think TypeScript...which has to be comfortable for the peanut gallery, while trying to support advanced features many know need to be there in a modern language, and the struggle between those 2 goals. That ensures we're going slower than we need to, while basically ensuring those techs will be obsolete sooner than later (because we need a stepping stone to the "right" solution as to not alienate developers)

gmarx 10y ago
Came here to make your second point. Most developers are bad and think they are good. I'll add that most of the people who provide requirements don't understand software development and most developers refuse or are unable to understand the domain space and yet insist that the requirements writers just tell them the "what" and leave the "how" to the developers. Oh- and using a process name, such as "agile" as an excuse rather than a constraint.
clifanatic 10y ago
> Most software engineers are bad and refuse to admit it to themselves (which would let them get better) Well… not to be confrontational, but in the context of this post, you seem to be implying that if most software engineers (which would, statistically speaking, include me) were “better”, they could put software together faster. I have to say, the implication bothers me, as does my suspicion that it’s a sentiment shared by most non-programmers out there. The problem is, there is a lower limit on how long it can possibly take to complete a programming task - and I think even the MBAs will concede that that lower limit is higher than the time spent typing out the words that make up the program. So, if we accept the premise that “good programmer” = “fast programmer” with no reasonable, objective measure of how fast fast is, the people doing the judging will always end up using the metric “how long I wanted it to take”, which is usually about a day for any task.

jbb555 10y ago

People STILL confuse the construction of software with the construction of buildings. We can estimate fairly accurately how long it will take to build a building once we have reasonable plans for it. I can pretty accurately say that it will take about 4 minutes to build the software once I have the plans to build it. The compiler pretty much automates the whole job. Writing software is NOT construction. Much of it isn't even design. Most of it is gathering detailed requirements and writing them down in unambiguous form (code). My asking how long it's going to take to write a software it's like saying to a building contractor how long will it take to design every single detail of a city block including gathering all the requirements. Also the requirements for software are much more detailed than building. 100000 lines of code represents 100000 decisions. I bet not many buildings have 100000 decisions. And 10000 is tiny for a software project.

yannickt 10y ago
There is a great essay by Jack Reeves making a similar point. What Is Software Design? http://www.developerdotstar.com/mag/articles/reeves_design.html http://www.developerdotstar.com/mag/articles/reeves_design.h... FTA: There is one consequence of considering code as software design that completely overwhelms all others. It is so important and so obvious that it is a total blind spot for most software organizations. This is the fact that software is cheap to build. It does not qualify as inexpensive; it is so cheap it is almost free. If source code is a software design, then actually building software is done by compilers and linkers.
- someguydave 10y ago
  But then you could argue that the machine code emitted from the compiler is a design and the actual hardware that implements it is "building the software".
  - someguydave 10y ago
    But then you could argue that the implementing transistor arrangement is a design and the actual movement of electrons that implements it is "building the software"

Futurebot 10y ago

- Requirements that change mid-stream, dozens, sometimes hundreds of times - Team has to do speculative development for ill-formed or understood problem domain - Requirements that are improperly gathered - Requirements that are underspecified - Poorly understood problem in general - Fighting bugs in system libraries - Fighting bugs in third party libraries - Fighting issues with integration on deployment and/or third party anything - Complex underlying technology requiring its own kind of discovery (like in the OP) - Poor/incorrect documentation - Project members that go off on their own, refuse to communicate, deviate wildly from the project style, break things - Management that insists on doing things from scratch - Management that insists on using buggy, broken, etc. 3rd-party or even in-house systems/libraries for political or faux-business reasons - Constant software developer interruptions - Long meetings that consist of circular discussions, politics, back-and-forth arguments about personal preferences disguised as "important project stuff" - Administrative minutiae that never stops adding up - People leave a project partway through, taking project knowledge with them - Broken/buggy tools/environments/systems - People getting pulled off a project to work on another project - Bad use of project management, source code management, other tools - Insistence on wheel reinvention for well-understood problems for career advancement, resume building, or boredom-relief purposes - Low morale / poor focus due to bad management, co-workers, bad tools, bad work environment - Low morale / poor focus due to reliance on: late night work (leading to insufficient sleep), weekend work, little/no time off or breaks, reliance on crunch/death marches - Dealing with company/project politics in general - Lack of focus due to team member resume-polishing/interviewing part way through a bad project - Occasional project sabotage

majewsky 10y ago
Hello, Mr. Brooks. Didn't see you coming in.

ausjke 10y ago

Maybe most programmers are amateurs? If you're really good at data-structure/algorithms/etc and practice them like a surgeon frequently, maybe the landscape will be very different. "Anybody can code, even an idiot can learn to code in 24 hours"

njharman 10y ago

Cause I (we?) spend too much time reading articles like this instead of coding.

Pica_soO 10y ago

Because a lot of stuff gets written several times. Doesent matter wether its agile or waterfall, and i dont care who started it - customers or software-engineering, wether the rewrite happened before roll out, or while the thing was already in the wild. Software development takes so long, because we tear down what we build and redo it, alot of times over.

deleted 10y ago

[deleted]

jimjimjim 10y ago

Following on the analogies already presented here: Coding is not construction (that's the compiler). Coding is the architect/drafter. If you want a pre-planned house, no problem, that's the same as buying shrink wrap. If you want to add a swimming pool under then that a whole bunch of conversations, customization and checking. If you want a completely bespoke house then then that is a conversation that lasts for months and the requirements will always change. Think about that Grand Designs tv show. that is software and the architect is software development

imichael 10y ago

To me it all boils down to this: hardware is all about making billions of things, all identical. Software is about building innumerable things on top of that hardware, all different. What would you like it to do, what can be done, how to do it, how to organize a group of people to do it, how to make sure it works and stays working, it's all creative work and it takes time to do it well.

drinchev 10y ago

Once I outsourced my side-project, since the pay per hour ratio worked better for me, compared to what I earn, combined with the complexity of that project. The "only" thing I had to do is write specs. The specs took me 3 weeks to write ( finding a lot of logical "bugs" in my project ), but the detail I went to ( SQL schema suggestions, framework suggestions, front-end plugins suggestions, ready pixel perfect design with different states, urls for the different pages, etc. ) was one of the biggest benefit for the final price. The company that took the project estimated 4 months and delivered it in 3 and half. They said those were the best specs they've ever seen. Moral of the story : Do full specs before you start coding ( I know it's boring ).

VladimirGolovin 10y ago
I did the same for my outsourced side-project - I went full waterfall. The spec turned out to be 250 pages long (without pics) and took almost a year to fully develop from the conception stage. In addition to this, I hired a designer to develop an additional design spec for the UI. So far, this worked great. My total time spent on this project since the moment I gave the spec to developers is about one workday in total. The questions from the devs are mostly trivial, and solvable in an hour or less.

keithnz 10y ago

my real takeaway from this article is the amazing feat of typing at 800 characters a minute! according to http://smallbusiness.chron.com/good-typing-speed-per-minute-71789.html http://smallbusiness.chron.com/good-typing-speed-per-minute-... a good typist does around 335 cpm so this is way over twice as fast is this a typo and supposed to be 80? that would be very slow.

Nadya 10y ago
My guess is that is with "bursting" and is not very sustainable. Not to mention, being able to type so quickly is very impractical when you need to think about what to say (well....type) and with programming especially, typing speed is irrelevant. To give a comparison - I can type 700~ cpm steadily, which means I can maintain it for large periods of time. Only practical when copying text when OCR fails me or when doing data entry. With "bursts" (unsustainable but rapid typing) I can type 160-170wpm with a negligible error rate (<1%) but I can only keep it up for a few minutes. Realistically, I type closer to 110-120wpm, because I take longer pauses between words while I consider what I am about to say (again....type). My hunch is the author actually meant 800cpm, but I do have my doubts to the accuracy or susustainability of the figure. It isn't "impossible" but it is "The top 1% of the top 1%" levels. :) http://i.imgur.com/NW2Kq4p.png http://i.imgur.com/NW2Kq4p.png

oneplane 10y ago

Because software is a half-visible moving target, that's why.

prefect42 10y ago
I like your answer the best, quite succinct. Software is / has way too many intangibles, by its nature. If you try to show the customer some tangible progress every three weeks, that's all very good, but strongly highlights the "intangiblesness" (sorry made up word) of it all. Once the customer starts seeing some "real" feature take shape, then the feature requests really start pouring in, and any previous understanding of this feature before this point, becomes a dim ephemeral dream in everyone's minds. Constant moving target.

deleted 10y ago

[deleted]

stevefeinstein 10y ago

It doesn't. It takes as long as it takes. It's a constantly evolving moving target. Why does writing a book or making a movie take so long? It's a creative thing. It's hard.

dockd 10y ago

I suspect it has something to do with a) The programmer not actually using the software they create. They end up spending time trying to understand the domain. b) The programmer uses the software they create and wants it to do everything (see also Second System Effect, Zawinski's Law/Law of Software Envelopment). c) The programmer expects components to work as documented.

deleted 10y ago

[deleted]

eagsalazar2 10y ago

Buildings are totally different in that they are all, roughly, the same or small variations on the same design. Consequently it is very easy to reuse massive amounts of reusable components in extremely predictable ways and to define standards for review that can be broadly applied across the majority of projects. This is not the case when builders are doing something that is actually totally novel in that they don't have many existing examples and it is built of entirely custom components. In cases like that it is actually exactly like software in that there is a lot of figuring things out as you go along, schedules are usually laughably off by orders of magnitude, and there are frequently major screw ups (Big dig, Seattle big bertha fiasco, SF Bay Bridge, ... to name a very few) Of course in software we aren't always reinventing everything as people unfairly complain. We reuse way more than we reinvent - postgres, elixir, phoenix, browsers, OSX, etc, etc, etc represent 1000s of lifetimes of work and the majority of the totality of work behind an actual product. The time we do spend, and consequently the variability in schedule, is due to the fact that most software is actually quite different in its requirements from all previous software and usually previous examples (aka competitors) have not made source code available for reuse. One huge difference between software development and "custom" building development is that good software developers embrace this variability and uncertainty - hence agile. Most other fields of engineering are still caught in an archaic and delusional world of gantt charts and hard deadlines with feature lists developed by architects. It never works that way insofar as the work is _custom_ - in any field! The thing I really hate about this conversation is how so many people are so eager to just chalk all this up to software developers being half-assed hacks. Well there are lots of teams at places like HP staffed by only "certified" software engineers and guess what? They still are terrible at predicting outcomes and they are actually far far worst in general at their jobs than small, modern, agile teams who make the best of the chaos by addressing that reality in how they work (again, agile). This is software development people. It isn't about bad devs and lack of certification, it is just hard. For comparison I did integrated circuit design for 7 years right out of school up through being a lead on multiple larger teams before switching to software. The rigor you perceive in other areas of engineering is a total fantasy. Bridges and buildings are maybe the exception but again this is only possible because it is such a canned art by comparison. If you disagree please just enumerate some specific "building codes" or "certification requirements" that you think would actually make any difference on your average complex software project.

ilaksh 10y ago

Why does it need another thread if ALSA uses poll()?