9 ms·

I'm Sure It Will Only Take You A Few Days To Code

221 points by dshipper 15y ago

tnash 15y ago
Thanks for this, it's a great article. I have this problem with my boss occasionally, which is funny because he's a programmer. "Oh, you just have to do this and this, shouldn't take you too long." You're right, there's always those bottlenecks that crop up that you didn't expect. It's impossible to truly know all the limitations of the software you're working with. I work with Drupal, and it has A LOT of those problems. Sometimes I'll think something will be easy to do, only to find out that the popular modules I thought would work great don't cater to the specific use-case I'm working on. Then it's back to schlepping.
- quanticle 15y ago
  As bad as it is for non-technical managers to estimate, I've found that in some cases, programmer managers can be worse, especially if you're new on the team. Why? They already know the code, so often their estimates state how long they would take to implement the feature. They either forget or don't realize that you have to spend additional time to learn how the code works and how to best integrate your change. They don't realize that sometimes it's harder to modify software than it is to build anew.
  - dshipper 15y ago
    Really interesting point here. Although I'd like to point out that I don't think it's necessarily being technical vs non-technical that makes it hard to estimate. There are probably non-technical people out there who have managed a lot of software projects and have developed solid intuition for how long projects will take. The real divider is experienced vs. inexperienced. But that's just a guess - I've never been managed by someone non-technical.
  - tnash 15y ago
    Very insightful, that's my case I believe. I had never used Drupal prior to starting at the place I am now, and part of every project so far has been finding out that Drupal does not, in fact, do what I want. Luckily my boss is a good guy, so he's understanding when I don't estimate correctly.
- abibibo 15y ago
  One of my favorite quotes from G.E.D: Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law http://en.wikipedia.org/wiki/Hofstadter%27s_law http://en.wikipedia.org/wiki/Hofstadter%27s_law
zeteo 15y ago
Funny to see this on the same front page as http://news.ycombinator.com/item?id=3795064 http://news.ycombinator.com/item?id=3795064. "[Jobs] pushed Steve Wozniak to create a game called Breakout. Woz said it would take months, but Jobs stared at him and insisted he could do it in four days. Woz knew that was impossible, but he ended up doing it."
- joe_the_user 15y ago
  Yeah and Linus coded git in a weekend... The infuriating thing for me is that people don't get that these monumentally fast development achievements still involved a massive expenditure of personal effort. Thus the "It'll take a few days of coding" customers of the gp are only more annoying if they cite examples like yours, expecting a complete product in a few days.
  - zeteo 15y ago
    Don't get me wrong, I'm a software engineer.
  - bigiain 15y ago
    Not only that, but they require all the skill and experience of truly world-class experts in their fields. Yes, Linus can throw together an amazing dvcs that scratches his particular current itch in a weekend. The just-graduated CS major you're pitching your idea to? The one with three whole php and/or ruby websites under his belt? He'll be able to do that too - after spending 20 years running the worlds most popular open source project and knowing exactly what he needs his version control system to do. Give him a weekend to do it right now? You'll get _something_, but it sure as hell wont be git. Same with Woz - there was probably nobody else on the planet who could have written breakout for the AppleII in under a month. I don't know whether Jobs was brilliant enough to know what he was asking was possible for Woz, even when Woz told him it wasn't, or whether Jobs just strongarmed Woz with unrealistic expectations and took credit for the success (and laid blame for any failure); but if he'd asked anybody else, they would have failed. Whatever your idea is, chances are _very_ high that if you think it could be coded "in a few days", you don't actually understand the problem (and it's solution) yourself.
noonespecial 15y ago
You are in a closet, looking at a closed door. How big is the house?
- andrewflnr 15y ago
  I hope I can remember to use that one.
- MattBearman 15y ago
  I wish I'd thought of this analogy when I was working for a big development agency.
- kitsune_ 15y ago
  I'm definitely going to use this one some time in the future.
- EliAndrewC 15y ago
  The more I think about it, the more I like this metaphor. Because you CAN make SOME estimates based on what's in the closet. If it's a huge closet, you could guess that the house was fairly large. If there are a lot of coats of different styles, you might infer that many people live in this house. But you're still making a guess about something you can't actually see because you haven't been given enough information. Brilliant!
- tlianza 15y ago
  The problem is no one has locked you in the closet - just open the door and look around. Heck, you have a whole hour to look around the house and estimate how long it takes to build the thing. You are limited by how long you can spend inspecting every nook and cranny of something, but you're only limited by time. The "door" metaphor is misleading. Nothing is standing in your way from exploring the space. If you can find the "hard parts" of the house faster than someone else, you're likely to be a better estimator. In fact, my favorite trick is to focus on one hard part and extrapolate up. For example, write down how tricky it is to build the kitchen, or heck just a cabinet in the kitchen, so everyone gets it when you say "oh, by the way, there are 10 more rooms."
ojbyrne 15y ago
A thought that occurred to me - another metric we use in the real word would be what I'm calling "solidness" or "weight." I.e. if you see a house and a similarly sized tent, the apparent difference we see between the two is weight - we can probably pick up the tent and walk away with it. In a sense, that's a useful way to describe software to non-technical users. We can throw something quick and dirty together, but it would be light as a feather, and carried away by any kind of load whatsoever. Or we can build something more solid to last longer, but it should be clear that that will take more effort.
- sirmarksalot 15y ago
  Yeah, let's just go with the quick and easy version. It's really all we need anyway, I mean it's not like we're going to release it as a product to the public. We just need a demo for the boss right now. <three months later> Why is that thing not working? I thought we finished it three months ago!
mc 15y ago
Nice write up. Want to add two other reasons we're horrible at making estimates. 1. Expectations. 2. No widely accepted language for writing requirements. Re: 1. Expectations are just hard to manage. Even when you do get "So the site's pretty simple" - the set of features in their brain is totally different from the set of features in your brain. Alas, it'll take more than "all it needs to do is X, Y, Z" to fix this process. Re: 2. Is there a professional software consulting company that has solved this problem? I say there isn't. I also say that with the advent of Agile/Lean Startup methodologies, it'll probably never happen - "No, we don't write reqs, we have stories you need to approve." uh huh, okay. For writing requirements, I got really damn fond of Cucumber, but then the lean-ux took off, and that shifted everybody's priorities. Bottom line, until there's a widely accepted way to communicate design and functionality, making bad estimates will always be easy to do
hkmurakami 15y ago
Disclosure: I am currently a software PM I actually think that the same principles apply to _any_ form of creative activity. Let's say I'm going to write a 300 page book. I can take my typical wpm speed, do some math on it, and say that it will take me X hours to do this. A book, on the surface, should fit into the "size and speed" heuristic. But of course, this is nowhere near the case. No decent piece of writing ever comes out without many many iterations of rewrites. Writing a piece of music? Same thing. Making a clay pot? Even great masters take many tries before coming up with something worthy of the public eye. Software, even to my novice eye, is similar. But the difference is that the person with zero experience doesn't know instinctively that software also falls into this iterative, creative process. We've been trained and educated to know that writing, composing, and sculpting are iterative creative processes that don't have a linear measurement scheme. We spent hours upon hours as children and young adults writing and rewriting, drawing and redrawing. We know from personal experience that things hardly go "as planned". But for something completely foreign to us, such prior knowledge is absent. We have no idea. So we take what we can see, and extrapolate. "Hey, I see just one web page, that can't be so hard... (insert mysterious thought process) how about XYZ days?" The fact that I took a few years of programming courses in college (I was an EE Major) has probably helped me in keeping things sane. I've written enough code myself to know instinctively that both development and debugging are far from linear processes. I know that I have no way of estimating how much time it will take devs who are much more capable than I am to develop/fix something. So I cede to the guys who know best. I take their word and estimates in good faith, and focus on swatting the 'flies' away so the team can focus on the task. What if we don't hit our estimates? Well too bad, but stuff happens. Time to reassess, buckle down, and go again.
- dshipper 15y ago
  I agree completely but I think there are two things that make software slightly unique from any piece of art: 1. We have a cultural context that tells us how hard it is to produce something like a song, or a painting. There's a reverence for it. And so people tend to back off of their estimations for the amount of time required to produce it. 2. Software isn't physical. You can hold a big book in your hand say CRAP! that must have taken a long time to do. But you're just looking at the result of a program, if it's done well, it's a tiny streamlined user interface. The result of one button click could have taken thousands of programming hours and thousands of lines of code but you would never know it. Software obfuscates complexity from the user in a way that someone looking at a huge huge painting or a big book doesn't run in to as much. So if we're talking about the efficiency of natural heuristics in estimating things, I think software is still slightly more difficult to estimate than art. But I agree with your overall point.
jgamman 15y ago
so reverse the question - "why does programming small things take so long?" (assuming x, y and z were relatively standard). is there a niche in this somewhere that maybe some % of the guru class should be focusing on 'making typical, trivial'? ie, we found 70% of websites include these features - download this template, push run and you have a website running. change text on front page manually might actually be good enough for 20% of the population...
- ams6110 15y ago
  You're describing Squarespace and a dozens of other offering like it, as well as any CMS.
- bigiain 15y ago
  Because "x, y, and z" are "standard" only in the "the great thing about standards is; there's so many of them to choose from" sense of the word. Recent example. Client: "We just want an online shop" - great clickety click, Magento is installed, stick their logo in, tweak the stylesheet to match, call it "done". Over the next few weeks/months: "Where's the blog?" "How do I enter auctions?" "I want to display products with individual pricing, but sell in cases of a dozen." "My SEO consultant says I need some 'landing pages' - where do I make them?" "This doesn't sign people up to out MailChimp email lists!" "It doesn't work right on my wifes iPhone." Discuss needs with client, install various plugins, a wordpress blog, tweak the templates to make single-pricing/bulk-selling "work" for boxes of 12 (and kick myself knowing it'll bite me in the ass one day doing it this way…) Then: "How do I make the auction extend the time if someone bids in the last minute?" "How do I enter products that come in boxes of 6 or 8 instead of 12?" "It works on the iPhone now, but not on a friends 2 year old cheapo Android 1.8 phone!" Then: "How do I auto calculate freight costs based on both weight and cubic size, for orders with multiple products?" (at which stage I start explaining combinatorial complexity and the knapsack problem to the client, and run a warm bath and remind myself "down, not across…")
  - dennisgorelik 15y ago
    So good strategy would be provide customers with almost free standard solution and then customize it step by step for ever increasing fees, right?
robertskmiles 15y ago
Another thing which I think kills accuracy in estimating development time is the ratio of time needed for "figuring out what to do" and "actually doing it". If you're learning a song, the first stage is very quick. You think about the problem for a moment and you decide on a plan like "Read the sheet music and try to play the song. Focus on areas you have trouble with. Repeat". The bulk of the time taken is spent "actually doing it", not "figuring out what to do". The thing about the "figuring out what to do" stage is that often in order to estimate the time needed for that stage, you basically have to have already completed it. The answer could come to you out of the blue, it could take weeks of thinking into dead ends. And with software, that first stage can be the bulk of the time. Software is a more like writing a novel than learning a song. The thing that takes longest isn't the actual writing, it's the figuring out what to write. If someone asked you how long it would take you to write a novel that gives insight into the human condition by following the story of a family caught up in a bloody civil war, what time estimate would you give them? I expect your typing WPM wouldn't be a big part of the calculation. You only know how long it's going to take once you know how you're going to do it, and once you know how you're going to do it, you've already done most of the hard work.
alaskamiller 15y ago
Same fallacy that convinces people to think they can change the world as young whippersnappers. I'm sure it'll only take you one life time to change the world. How you do guess at something without knowing the full size, weight, depth, and complexity of something?
edw519 15y ago
The Lifecycle of a Project Manager: Phase I: (Idealism) Treat building software like building a bridge. Analysis --> Design --> Development --> Testing --> Implementation. Phase II: (Pragmatism) Realize that Phase I doesn't work. Try to figure out why. Decide that the weak link is Analysis, i.e. we're no good at estimating and hitting deadlines because we never have good enough specs. Devote your life to the art and science of getting the perfect spec. Implement the methodology du jour to make that happen. Phase III: (Realism) Realize that Phase II doesn't work. Decide that no existing Analysis methodology works because no one knows how to conduct analysis, no matter what the methodology. Focus on optimizing the practitioners with better education and 10,000 hours of real-world experience. Phase IV: (Enlightenment) Realize that none of the other phases works because we've been focusing on the wrong constraints all along. Instead of optimizing scope, quality, or quantity of the product, optimize the schedule. Adopt a new model: define the deadline first, then build whatever you can by that deadline. Hit your deadlines every time, not by estimating better, but by not caring how pretty your deliverable is. Eliminate the problem of estimating by making the deadline a constant. Eliminate the problem of analysis by replacing it with prototyping and revising. And eliminate the problem of blueprints by adopting a strategy of inventing, not engineering.
- SkyMarshal 15y ago
  Nice. You should make that a blog post so we can send the link to all newbie PMs we encounter.
- Bamafan 15y ago
  " Hit your deadlines every time, not by estimating better, but by not caring how pretty your deliverable is." I'd kill for some Phase IV PMs. :) (Actually, it's rare that I meet Phase I PMs. Very few PMs think in terms of developing software across those 5 stages. To them it's Photoshop -> Make Photoshop Mockup Do stuff -> Fin)
- kamaal 15y ago
  Most managers would know Phase IV if at all they had done some real work. As it happens people generally go over to Project Management if they can't do technical work. This creates a problem, you are asking somebody to manage things which he himself didn't want to do.
wildmXranat 15y ago
Let's take it a step further, shall we. Say you get this type of request thrown at you, which is obviously a mis-estimate and you don't end up delivering it with expected pace. You weren't supposed to, by the way. It was a trap. Now, the PM or whoever says this "Well, Joe, what if we punt it to Bob over there and see if we can get you concentrated on another problem X". Does that precipitate your balls into shrinking or what? Estimates are just that, estimates and not contracts. We should expect them to have a float time, otherwise rename them to contracts. I do X in Y amount of time if Z is true. Usually, there are unknown variables and I'm running out of letters to name them all. Furthermore, I don't work on a deserted island. I have a phone, QA keeps on asking me questions, clients want clarifications and my version control is burning up from constant merging, and yes I have included all that in the time estimate.
ams6110 15y ago
As a programmer I know I can use my prior experience building similar things to estimate how long each feature will take to implement. Here's where this breaks down for me: it's never similar. Sure, the functionality from the user perspective might be similar, but with the pace of change in development tools, platforms, and frameworks, it's never the same. Seems that every project involves some major piece of kit that I've never used before. In the last five years I've used ASP, ASP.NET, C#, PHP, Lua, Python, Erlang, Oracle and PL/SQL, MySQL, PostgreSQL, SQL Server and TSQL, OpenLDAP, XML, XSLT, JSON, javascript, jQuery, Scriptaculous, and other things built out of those, and probably a dozen others I'm forgetting at the moment. Yeah at some level databases are all basically the same and languages are all basically the same but at another level they are not and that's where the time sinks are.
andrewfelix 15y ago
I think a lot of clients offer estimates based on how much money they have to spend regardless of the complexity.
- kitsune_ 15y ago
  No shit. It's always like this. They have a wish list and a budget. These two things hardly ever match. It's hard to give a quote when there are no detailed specs yet. Most clients still want a price tag. Of course afterwards they'll nail you on this premature conjecture. As a business your quote needs to be competitive and still be profitable. From experience, we usually throw things out and create boundaries in our offers / quotes. We've learnt the hard way that often times clients implicitly assume certain features to be there without ever communicating them to us, so you need to think ahead and have it in black and white that if they want X they have to get another offer / quote at later stage. So imo it's not just about what work a quote includes, but also what work it does not include.
6ren 15y ago
Reminds me of "open textured" in legal philosophy: the idea of precedent in law is you can use it to predict how a court will decide in future. This mostly works, but if you have a bunch of cases (data points in multidimensional space), simple interpolation isn't always accurate. You need to look at the specifics of the present case (there's still uncertainty because each judge has an individual perspective, but that's a different issue). It's a little bit like a fractal: just knowing some data points won't always predict - even roughly - what happens in between (but in flatter regions, it can). And Turing showed that (in general), you can't predict the result of a computation except by actually doing it. The experience the article mentions only helps when a specific domain reoccurs, and known tools have been developed for it, with known characteristics. Now, if you know that domain and the tools, you can do prediction pretty well. I think of it like an experienced tradesman. However, when you go into a new area (and a new one is created every decade or so in programming), you're back to zero in many respects. e.g. the existence of the right library can save 10,000 hours.
- gruseom 15y ago
  That makes me wonder what other disciplines there are that have the unpredictability and open-endedness of software. I read that on large construction projects, a 1% overrun is considered failure. That is clearly a different universe. Who else is in our universe?
  - 6ren 15y ago
    I think it includes anything involving new knowledge: e.g. law, science/mathematics, invention. Maybe also in the academic Arts (I don't know enough to be sure), and also in the creative arts - painting, sculpture, writing (there's exploration and massive overruns there). Also, exploration in prospecting, oil discovery - though that's not really open-ended. OK, I think it's being able to open up/zoom into finer detail (like knowledge, recursive structures, fractal) + discovery.
    - gruseom 15y ago
      Law, I'm not so sure. Any variations posed by new legal cases are still limited by the basics of human nature. Oil discovery? Don't think so. People may look in different places but they don't invent new ways of doing each project. The other examples are all art. (The practice of science itself is an art and not a science.) Is software an art?
mkramlich 15y ago
non-programmers, in my experience, tend to think software is "just that stuff you see on the screen". So if they create some screen mockups of the UI, they think the programmer pretty much needs to sit down at the computer, and do some simple keyboard/mouse clicky-clicky to make the screen match his mockups. and then they're done. To them, it's just visual, and static. They don't understand there are processes, threads, hardware contraints (cpu, memory, disk, network bandwith), libraries, languages, configuration, edge cases, error conditions, etc. I've seen this kind of perspective a lot with non-technical stakeholders, especially first-time clients of software contractors, or first-time managers of software developers. "Just make it match these mockups!" Dragging a little Facebook icon into place and poof all the code and edge case handling and database work needed to integrate with Facebook is all done, put to bed, etc. Just put that icon in the right place.
- kamaal 15y ago
  Well in such a case those people should not opt to manage technical people.
robomartin 15y ago
P = programmer; NP = non-programmer NP: How long will it take? P: It'll take as long as it takes. NP: I don't understand. What does that mean? P: When you tell me that the product is finished, it's done. NP: You mean you can't estimate how long it'll take? P: I can't estimate when you'll decide that it is done, therefore an accurate prediction is impossible. From experience I know that a project like this can take from three to nine months. A lot depends on you, the choices you make and how well you communicate them to me. A lot also depends on how many times you change your mind during the process and how many times you change the specifications during the job. Major changes can even cause the project to be scrapped and start over. Rest assured that I will not be the one slowing this down. NP: How much will it cost? P: It will cost as much as it costs. NP: I don't understand. What does that mean? P: You just learned some programming! Go back to your first question and re-read the entire thing. This is called an endless loop.
- bravura 15y ago
  What are the implications on the universe if P=NP?
  - deleted 15y ago
    [deleted]
- buu700 15y ago
  The real question is, are they the same person?
- doktrin 15y ago
  This takes a fairly narrow view of a development process, IMO. The "but you didn't specifically tell me to do X" attitude can border on the irresponsible and/or lazy. It's all nice and well to expect accurate step-by instructions to be communicated by a PM, but the onus also does fall on developers to meet them half-way. While machines may require explicit and literal step-by instructions, a human being value-add is in thinking rationally, making inferences and (when necessary) asking questions.
- 15y ago
robomartin 15y ago
One of my favorite ones was a fellow who wanted an iPhone and iPad game done. He proclaimed that it had to be done for $10K. This game easily required six months of work. It also required server-side development and support. He also had no idea that every so often you need to fix, I mean, update, your apps because new iOS releases might break them. I was open-minded and decided to invest some time educating him on the process and the needs of the platform. Once he had enough information he networked and hired someone out of India to build him the game. I was not happy about that at all. Then I learned that he was having all kinds of problems with the process (and the app) being a total mess. He's learning his lesson.
- yardie 15y ago
  Come on now. Don't blame the programmer being Indian for the problems your friend is having. Good programmers anywhere cost almost as much as they do in the US now. Globalization means that an Indian programmer has the same access to salary information in the west. He/she might knock off 10% to be competitive. You're friend could have easily got a cheap, inexperienced programmer anywhere else in the world, including the US, and still had the same problems. He had a budget, you knew it wasn't enough for what he needed, and he came up snake eyes.
  - dpritchett 15y ago
    If an unsophisticated client looks to another hemisphere to get a one-off iOS app built I wouldn't bet "world-class talent at a 10% discount" was their hiring strategy. More likely "I can get this done for 1/3 of what this guy is quoting me? What could possibly go wrong?"
  - robomartin 15y ago
    Who says I am blaming the programmer for being Indian? I'm not. Excellent work being done out of India. Here's the problem, lots of people like this fellow look at India and China to find the lowest possible bidder out of complete ignorance. And, when they find the lowest possible bidder they get exactly that: crappy inexperienced programmers. If they searched for the lowest possible bidder in the US or Europe they'd get exactly the same thing. I did not intend to imply that Indian programmers are not good. An ignorant fool looking for rock bottom prices will, more than likely, find them in India rather than the US or Europe.
b1daly 15y ago
I'm a music producer, and this is a constant problem for me that after many years still boggles my mind. The root of the problem is that producing a modern style recording of music is resource expensive. Far more than anybody would like it to be. The resources can be time/money/talent. So there is an intense bias on everyone involved in the project to under-estimate. I usually work on low budget projects. You can actually accomplish a lot with a low budget. But as you start to climb the "quality" scale the costs increase much more quickly than the quality. To get a product that you might consider subjectively twice as good might cost ten times as much. At core, I feel like there is a fundamental principal of information organization. It takes a certain amount of resources to undo entropy and organize information. When we experience a piece of work we can intuitively feel that entropy has been significantly reduced, we appreciate the accomplishment. (Even if we are not in the field of endeavor we still feel it though we don't know how it is done.) One specific point of difficulty in estimating time in music production is that there is an x-factor. It derives from the need to create something exceptional, which has both subjective and objective elements. With an experienced production team you can predict how long it will take to produce something competent. But usually there is a point where we have all done the things we know how to do: the song is arranged, the players are good, the production environment is good, the instruments are good. But the song is missing something. What is it? It is not known yet. So an iterative trial and error process begins. Let's try a different singer. Maybe we should borrow my friends guitar. Let's keep recording. Let's rewrite the song. Hopefully a solution is arrived at, but since it is a trial and error process it is impossible to estimate how long it will take.
jtheory 15y ago
Is Pachelbel's Canon considered complicated as far as music performance goes? Reading that knocked me right out of the flow of this article. What's he thinking of? Maybe someone's complicated improvisations on top of the (really simple) canon? On topic: the post didn't really cover where the layperson's base estimate comes from -- a layperson sees X pages, or X features, and multiplies that by... what? The reference they use varies hugely, in my experience; many are trying to imagine what's involved in developing a "feature" -- which yes, will generally be a massive underestimate... not always, though. There are also plenty of laypeople who imagine that everything they see is incredibly complex & difficult to create (I have a neighbor who imagined that writing software meant typing in the gunk he saw when accidentally opening a binary file in notepad), and all written from scratch -- so a simple blog would take ages to build, because it's obviously complicated stuff. In real life, some quite complicated things come for free, other seemingly simple things take gobs of time, sometimes unexpectedly... it's a big mess. And just as some of the harshest wildcards (like nasty bugs in old browsers) are calming down, more form factors, interaction modes, etc. are popping up, so I'm not sure it's going to be a problem we can stop talking about anytime soon.
- damncabbage 15y ago
  Is Pachelbel's Canon considered complicated as far as music performance goes? Consider the more complicated violin part.
  - jtheory 15y ago
    Still, you hear grade schoolers play it. And wow, if you've ever been forced to play the part of anything other than violin -- I played the viola part innumerable times in HS orchestra, and the cello part was even worse -- it's almost a punishment.
NameNickHN 15y ago
Why does the way we naturally measure complexity stop working when we apply it to programming? Lots of projects that have nothing to do with programming end up taking longer than expected or are over budget.
jsvaughan 15y ago
"But it brings up another more interesting question: why does the way we naturally measure complexity stop working when we apply it to programming?" Ah and I had high hopes for that next paragraph, There are two fundamental differences between, lets say, estimating how long it will take to make a sandwich, and estimating how long it will take to create an online noticeboard for your village. The first difference is that one task is manufacture, the second task is design. Writing code IS design, and it is inherently much harder (or even logically impossible) to estimate because implicitly you do not know what done means. The second difference is that (broadly) how satisfactory a sandwich or other real world object is, is proportional to how long it takes to make. This is not true for software, because it is not a physical entity. It can be tripped up by a single incorrect character, and then, forget tasty - your software sandwich doesn't even exist.
scanr 15y ago
Any project management software out there that takes an ML / probabilistic approach to this? One thing I found at one of the mega-corps that I worked at is that both project managers and regular managers were blind to history. Note: this was before agile became accepted by mega-corps. Manager: How long is this going to take? Me: Virtually impossible to predict accurately Manager: I need a date Me: 3 months Manager: That's too long, we need to get it done in 1 month Me: OK, 1 month ... 5 months and several iterations later ... Manager: Project has been deprioritized, we have a new project, how long is this going to take? Me: Virtually impossible to predict accurately Manager: I need a date Me: I need you to be stateful.
Scotchy 15y ago
I very often hear this comment from my fellow developper colleagues. In our startup, I am responsible for the frontend coding. Very frequently, I get requests from other developpers to implement this or this, and very often those requests are annotated by a "c'mon, it is just CSS, it'll only take you 5 minutes" when they have absolutely no idea what they are talking about, because they never ever did real frontend developpment. They simply think that everything boils down to HTML and that frontend is NOT really programming. Attempting to explain that a frontend also has logic similar to backend, and that no, plotting graphs when you don't even have the data modeled is not just a 2 liner with Google Visualisation. I am always offended by people estimating time for work they never did. So I find myself very rarely giving estimations. Sometimes when I have a very precise idea on how to implement an idea right when I'm being told about it, I usually say "yeah this is definitely doable in a short time span". Meaning things that do not take more than half a day to implement, because you already compiled the code in your brain. Now, when a colleague of mine estimates stuff for me, I tell them "No I don't think this is going to take X minutes to implement. If you think it is implementable so fast, why don't you go ahead and do it.". When they take it to the word, and I see them still struggling after many hours on some problem that are trivial to me, I will go ahead and help them out. After a few times, they usually won't bother me with estimations anymore... NOT.
adrianhoward 15y ago
I think it's much more domain-expert vs non-domain-expert rather than programmer vs non-programmer. I've regularly seen very experienced developers make exactly the same sort of mistakes when talking to people outside of their field (Sales, Design, etc.) because they miss things like: * Sales is about building stories and relationships - it's not just listing features and meeting people. * Design is about understanding the user, iterating, testing and experimentation - it's not just making things pretty. I wish technical folk were naturally better at this sort of thing - but they (and I :-) keep making similar sorts of mistakes. (It's also amusing to see the "it's not like building a bridge" analogy turn up. Go talk to some engineers, architects and builders - and find out how many of those projects ended up matching the original time and cost estimates :-)
- DanBC 15y ago
  > (It's also amusing to see the "it's not like building a bridge" analogy turn up. Go talk to some engineers, architects and builders - and find out how many of those projects ended up matching the original time and cost estimates :-) That's an important point. I used to work in sub-contract electronic engineering. We had a lot of information about how long it took to do things, and how much items cost. In theory, estimating the cost of projects should have been straightforward. And then you add 15% for wiggle room. And still it's hard to keep things going out on time and on budget.
Heisens 15y ago
Such a noob topic. If you don't know what you are doing, just admit it. For seasoned programmers and PMs, this is a non-issue. Perhaps you are in the wrong profession if you can't handle the demands without whining like a brat.

221 points by dshipper 15y ago

tnash 15y ago

Thanks for this, it's a great article. I have this problem with my boss occasionally, which is funny because he's a programmer. "Oh, you just have to do this and this, shouldn't take you too long." You're right, there's always those bottlenecks that crop up that you didn't expect. It's impossible to truly know all the limitations of the software you're working with. I work with Drupal, and it has A LOT of those problems. Sometimes I'll think something will be easy to do, only to find out that the popular modules I thought would work great don't cater to the specific use-case I'm working on. Then it's back to schlepping.

quanticle 15y ago
As bad as it is for non-technical managers to estimate, I've found that in some cases, programmer managers can be worse, especially if you're new on the team. Why? They already know the code, so often their estimates state how long they would take to implement the feature. They either forget or don't realize that you have to spend additional time to learn how the code works and how to best integrate your change. They don't realize that sometimes it's harder to modify software than it is to build anew.
- dshipper 15y ago
  Really interesting point here. Although I'd like to point out that I don't think it's necessarily being technical vs non-technical that makes it hard to estimate. There are probably non-technical people out there who have managed a lot of software projects and have developed solid intuition for how long projects will take. The real divider is experienced vs. inexperienced. But that's just a guess - I've never been managed by someone non-technical.
- tnash 15y ago
  Very insightful, that's my case I believe. I had never used Drupal prior to starting at the place I am now, and part of every project so far has been finding out that Drupal does not, in fact, do what I want. Luckily my boss is a good guy, so he's understanding when I don't estimate correctly.
abibibo 15y ago
One of my favorite quotes from G.E.D: Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law http://en.wikipedia.org/wiki/Hofstadter%27s_law http://en.wikipedia.org/wiki/Hofstadter%27s_law

zeteo 15y ago

Funny to see this on the same front page as http://news.ycombinator.com/item?id=3795064 http://news.ycombinator.com/item?id=3795064. "[Jobs] pushed Steve Wozniak to create a game called Breakout. Woz said it would take months, but Jobs stared at him and insisted he could do it in four days. Woz knew that was impossible, but he ended up doing it."

joe_the_user 15y ago
Yeah and Linus coded git in a weekend... The infuriating thing for me is that people don't get that these monumentally fast development achievements still involved a massive expenditure of personal effort. Thus the "It'll take a few days of coding" customers of the gp are only more annoying if they cite examples like yours, expecting a complete product in a few days.
- zeteo 15y ago
  Don't get me wrong, I'm a software engineer.
- bigiain 15y ago
  Not only that, but they require all the skill and experience of truly world-class experts in their fields. Yes, Linus can throw together an amazing dvcs that scratches his particular current itch in a weekend. The just-graduated CS major you're pitching your idea to? The one with three whole php and/or ruby websites under his belt? He'll be able to do that too - after spending 20 years running the worlds most popular open source project and knowing exactly what he needs his version control system to do. Give him a weekend to do it right now? You'll get _something_, but it sure as hell wont be git. Same with Woz - there was probably nobody else on the planet who could have written breakout for the AppleII in under a month. I don't know whether Jobs was brilliant enough to know what he was asking was possible for Woz, even when Woz told him it wasn't, or whether Jobs just strongarmed Woz with unrealistic expectations and took credit for the success (and laid blame for any failure); but if he'd asked anybody else, they would have failed. Whatever your idea is, chances are _very_ high that if you think it could be coded "in a few days", you don't actually understand the problem (and it's solution) yourself.

noonespecial 15y ago

You are in a closet, looking at a closed door. How big is the house?

andrewflnr 15y ago
I hope I can remember to use that one.
MattBearman 15y ago
I wish I'd thought of this analogy when I was working for a big development agency.
kitsune_ 15y ago
I'm definitely going to use this one some time in the future.
EliAndrewC 15y ago
The more I think about it, the more I like this metaphor. Because you CAN make SOME estimates based on what's in the closet. If it's a huge closet, you could guess that the house was fairly large. If there are a lot of coats of different styles, you might infer that many people live in this house. But you're still making a guess about something you can't actually see because you haven't been given enough information. Brilliant!
tlianza 15y ago
The problem is no one has locked you in the closet - just open the door and look around. Heck, you have a whole hour to look around the house and estimate how long it takes to build the thing. You are limited by how long you can spend inspecting every nook and cranny of something, but you're only limited by time. The "door" metaphor is misleading. Nothing is standing in your way from exploring the space. If you can find the "hard parts" of the house faster than someone else, you're likely to be a better estimator. In fact, my favorite trick is to focus on one hard part and extrapolate up. For example, write down how tricky it is to build the kitchen, or heck just a cabinet in the kitchen, so everyone gets it when you say "oh, by the way, there are 10 more rooms."

ojbyrne 15y ago

A thought that occurred to me - another metric we use in the real word would be what I'm calling "solidness" or "weight." I.e. if you see a house and a similarly sized tent, the apparent difference we see between the two is weight - we can probably pick up the tent and walk away with it. In a sense, that's a useful way to describe software to non-technical users. We can throw something quick and dirty together, but it would be light as a feather, and carried away by any kind of load whatsoever. Or we can build something more solid to last longer, but it should be clear that that will take more effort.

sirmarksalot 15y ago
Yeah, let's just go with the quick and easy version. It's really all we need anyway, I mean it's not like we're going to release it as a product to the public. We just need a demo for the boss right now. <three months later> Why is that thing not working? I thought we finished it three months ago!

mc 15y ago

Nice write up. Want to add two other reasons we're horrible at making estimates. 1. Expectations. 2. No widely accepted language for writing requirements. Re: 1. Expectations are just hard to manage. Even when you do get "So the site's pretty simple" - the set of features in their brain is totally different from the set of features in your brain. Alas, it'll take more than "all it needs to do is X, Y, Z" to fix this process. Re: 2. Is there a professional software consulting company that has solved this problem? I say there isn't. I also say that with the advent of Agile/Lean Startup methodologies, it'll probably never happen - "No, we don't write reqs, we have stories you need to approve." uh huh, okay. For writing requirements, I got really damn fond of Cucumber, but then the lean-ux took off, and that shifted everybody's priorities. Bottom line, until there's a widely accepted way to communicate design and functionality, making bad estimates will always be easy to do

hkmurakami 15y ago

Disclosure: I am currently a software PM I actually think that the same principles apply to _any_ form of creative activity. Let's say I'm going to write a 300 page book. I can take my typical wpm speed, do some math on it, and say that it will take me X hours to do this. A book, on the surface, should fit into the "size and speed" heuristic. But of course, this is nowhere near the case. No decent piece of writing ever comes out without many many iterations of rewrites. Writing a piece of music? Same thing. Making a clay pot? Even great masters take many tries before coming up with something worthy of the public eye. Software, even to my novice eye, is similar. But the difference is that the person with zero experience doesn't know instinctively that software also falls into this iterative, creative process. We've been trained and educated to know that writing, composing, and sculpting are iterative creative processes that don't have a linear measurement scheme. We spent hours upon hours as children and young adults writing and rewriting, drawing and redrawing. We know from personal experience that things hardly go "as planned". But for something completely foreign to us, such prior knowledge is absent. We have no idea. So we take what we can see, and extrapolate. "Hey, I see just one web page, that can't be so hard... (insert mysterious thought process) how about XYZ days?" The fact that I took a few years of programming courses in college (I was an EE Major) has probably helped me in keeping things sane. I've written enough code myself to know instinctively that both development and debugging are far from linear processes. I know that I have no way of estimating how much time it will take devs who are much more capable than I am to develop/fix something. So I cede to the guys who know best. I take their word and estimates in good faith, and focus on swatting the 'flies' away so the team can focus on the task. What if we don't hit our estimates? Well too bad, but stuff happens. Time to reassess, buckle down, and go again.

dshipper 15y ago
I agree completely but I think there are two things that make software slightly unique from any piece of art: 1. We have a cultural context that tells us how hard it is to produce something like a song, or a painting. There's a reverence for it. And so people tend to back off of their estimations for the amount of time required to produce it. 2. Software isn't physical. You can hold a big book in your hand say CRAP! that must have taken a long time to do. But you're just looking at the result of a program, if it's done well, it's a tiny streamlined user interface. The result of one button click could have taken thousands of programming hours and thousands of lines of code but you would never know it. Software obfuscates complexity from the user in a way that someone looking at a huge huge painting or a big book doesn't run in to as much. So if we're talking about the efficiency of natural heuristics in estimating things, I think software is still slightly more difficult to estimate than art. But I agree with your overall point.

jgamman 15y ago

so reverse the question - "why does programming small things take so long?" (assuming x, y and z were relatively standard). is there a niche in this somewhere that maybe some % of the guru class should be focusing on 'making typical, trivial'? ie, we found 70% of websites include these features - download this template, push run and you have a website running. change text on front page manually might actually be good enough for 20% of the population...

ams6110 15y ago
You're describing Squarespace and a dozens of other offering like it, as well as any CMS.
bigiain 15y ago
Because "x, y, and z" are "standard" only in the "the great thing about standards is; there's so many of them to choose from" sense of the word. Recent example. Client: "We just want an online shop" - great clickety click, Magento is installed, stick their logo in, tweak the stylesheet to match, call it "done". Over the next few weeks/months: "Where's the blog?" "How do I enter auctions?" "I want to display products with individual pricing, but sell in cases of a dozen." "My SEO consultant says I need some 'landing pages' - where do I make them?" "This doesn't sign people up to out MailChimp email lists!" "It doesn't work right on my wifes iPhone." Discuss needs with client, install various plugins, a wordpress blog, tweak the templates to make single-pricing/bulk-selling "work" for boxes of 12 (and kick myself knowing it'll bite me in the ass one day doing it this way…) Then: "How do I make the auction extend the time if someone bids in the last minute?" "How do I enter products that come in boxes of 6 or 8 instead of 12?" "It works on the iPhone now, but not on a friends 2 year old cheapo Android 1.8 phone!" Then: "How do I auto calculate freight costs based on both weight and cubic size, for orders with multiple products?" (at which stage I start explaining combinatorial complexity and the knapsack problem to the client, and run a warm bath and remind myself "down, not across…")
- dennisgorelik 15y ago
  So good strategy would be provide customers with almost free standard solution and then customize it step by step for ever increasing fees, right?

robertskmiles 15y ago

Another thing which I think kills accuracy in estimating development time is the ratio of time needed for "figuring out what to do" and "actually doing it". If you're learning a song, the first stage is very quick. You think about the problem for a moment and you decide on a plan like "Read the sheet music and try to play the song. Focus on areas you have trouble with. Repeat". The bulk of the time taken is spent "actually doing it", not "figuring out what to do". The thing about the "figuring out what to do" stage is that often in order to estimate the time needed for that stage, you basically have to have already completed it. The answer could come to you out of the blue, it could take weeks of thinking into dead ends. And with software, that first stage can be the bulk of the time. Software is a more like writing a novel than learning a song. The thing that takes longest isn't the actual writing, it's the figuring out what to write. If someone asked you how long it would take you to write a novel that gives insight into the human condition by following the story of a family caught up in a bloody civil war, what time estimate would you give them? I expect your typing WPM wouldn't be a big part of the calculation. You only know how long it's going to take once you know how you're going to do it, and once you know how you're going to do it, you've already done most of the hard work.

alaskamiller 15y ago

Same fallacy that convinces people to think they can change the world as young whippersnappers. I'm sure it'll only take you one life time to change the world. How you do guess at something without knowing the full size, weight, depth, and complexity of something?

edw519 15y ago

The Lifecycle of a Project Manager: Phase I: (Idealism) Treat building software like building a bridge. Analysis --> Design --> Development --> Testing --> Implementation. Phase II: (Pragmatism) Realize that Phase I doesn't work. Try to figure out why. Decide that the weak link is Analysis, i.e. we're no good at estimating and hitting deadlines because we never have good enough specs. Devote your life to the art and science of getting the perfect spec. Implement the methodology du jour to make that happen. Phase III: (Realism) Realize that Phase II doesn't work. Decide that no existing Analysis methodology works because no one knows how to conduct analysis, no matter what the methodology. Focus on optimizing the practitioners with better education and 10,000 hours of real-world experience. Phase IV: (Enlightenment) Realize that none of the other phases works because we've been focusing on the wrong constraints all along. Instead of optimizing scope, quality, or quantity of the product, optimize the schedule. Adopt a new model: define the deadline first, then build whatever you can by that deadline. Hit your deadlines every time, not by estimating better, but by not caring how pretty your deliverable is. Eliminate the problem of estimating by making the deadline a constant. Eliminate the problem of analysis by replacing it with prototyping and revising. And eliminate the problem of blueprints by adopting a strategy of inventing, not engineering.

SkyMarshal 15y ago
Nice. You should make that a blog post so we can send the link to all newbie PMs we encounter.
Bamafan 15y ago
" Hit your deadlines every time, not by estimating better, but by not caring how pretty your deliverable is." I'd kill for some Phase IV PMs. :) (Actually, it's rare that I meet Phase I PMs. Very few PMs think in terms of developing software across those 5 stages. To them it's Photoshop -> Make Photoshop Mockup Do stuff -> Fin)
kamaal 15y ago
Most managers would know Phase IV if at all they had done some real work. As it happens people generally go over to Project Management if they can't do technical work. This creates a problem, you are asking somebody to manage things which he himself didn't want to do.

wildmXranat 15y ago

Let's take it a step further, shall we. Say you get this type of request thrown at you, which is obviously a mis-estimate and you don't end up delivering it with expected pace. You weren't supposed to, by the way. It was a trap. Now, the PM or whoever says this "Well, Joe, what if we punt it to Bob over there and see if we can get you concentrated on another problem X". Does that precipitate your balls into shrinking or what? Estimates are just that, estimates and not contracts. We should expect them to have a float time, otherwise rename them to contracts. I do X in Y amount of time if Z is true. Usually, there are unknown variables and I'm running out of letters to name them all. Furthermore, I don't work on a deserted island. I have a phone, QA keeps on asking me questions, clients want clarifications and my version control is burning up from constant merging, and yes I have included all that in the time estimate.

ams6110 15y ago

As a programmer I know I can use my prior experience building similar things to estimate how long each feature will take to implement. Here's where this breaks down for me: it's never similar. Sure, the functionality from the user perspective might be similar, but with the pace of change in development tools, platforms, and frameworks, it's never the same. Seems that every project involves some major piece of kit that I've never used before. In the last five years I've used ASP, ASP.NET, C#, PHP, Lua, Python, Erlang, Oracle and PL/SQL, MySQL, PostgreSQL, SQL Server and TSQL, OpenLDAP, XML, XSLT, JSON, javascript, jQuery, Scriptaculous, and other things built out of those, and probably a dozen others I'm forgetting at the moment. Yeah at some level databases are all basically the same and languages are all basically the same but at another level they are not and that's where the time sinks are.

andrewfelix 15y ago

I think a lot of clients offer estimates based on how much money they have to spend regardless of the complexity.

kitsune_ 15y ago
No shit. It's always like this. They have a wish list and a budget. These two things hardly ever match. It's hard to give a quote when there are no detailed specs yet. Most clients still want a price tag. Of course afterwards they'll nail you on this premature conjecture. As a business your quote needs to be competitive and still be profitable. From experience, we usually throw things out and create boundaries in our offers / quotes. We've learnt the hard way that often times clients implicitly assume certain features to be there without ever communicating them to us, so you need to think ahead and have it in black and white that if they want X they have to get another offer / quote at later stage. So imo it's not just about what work a quote includes, but also what work it does not include.

6ren 15y ago

Reminds me of "open textured" in legal philosophy: the idea of precedent in law is you can use it to predict how a court will decide in future. This mostly works, but if you have a bunch of cases (data points in multidimensional space), simple interpolation isn't always accurate. You need to look at the specifics of the present case (there's still uncertainty because each judge has an individual perspective, but that's a different issue). It's a little bit like a fractal: just knowing some data points won't always predict - even roughly - what happens in between (but in flatter regions, it can). And Turing showed that (in general), you can't predict the result of a computation except by actually doing it. The experience the article mentions only helps when a specific domain reoccurs, and known tools have been developed for it, with known characteristics. Now, if you know that domain and the tools, you can do prediction pretty well. I think of it like an experienced tradesman. However, when you go into a new area (and a new one is created every decade or so in programming), you're back to zero in many respects. e.g. the existence of the right library can save 10,000 hours.

gruseom 15y ago
That makes me wonder what other disciplines there are that have the unpredictability and open-endedness of software. I read that on large construction projects, a 1% overrun is considered failure. That is clearly a different universe. Who else is in our universe?
- 6ren 15y ago
  I think it includes anything involving new knowledge: e.g. law, science/mathematics, invention. Maybe also in the academic Arts (I don't know enough to be sure), and also in the creative arts - painting, sculpture, writing (there's exploration and massive overruns there). Also, exploration in prospecting, oil discovery - though that's not really open-ended. OK, I think it's being able to open up/zoom into finer detail (like knowledge, recursive structures, fractal) + discovery.
  - gruseom 15y ago
    Law, I'm not so sure. Any variations posed by new legal cases are still limited by the basics of human nature. Oil discovery? Don't think so. People may look in different places but they don't invent new ways of doing each project. The other examples are all art. (The practice of science itself is an art and not a science.) Is software an art?

mkramlich 15y ago

non-programmers, in my experience, tend to think software is "just that stuff you see on the screen". So if they create some screen mockups of the UI, they think the programmer pretty much needs to sit down at the computer, and do some simple keyboard/mouse clicky-clicky to make the screen match his mockups. and then they're done. To them, it's just visual, and static. They don't understand there are processes, threads, hardware contraints (cpu, memory, disk, network bandwith), libraries, languages, configuration, edge cases, error conditions, etc. I've seen this kind of perspective a lot with non-technical stakeholders, especially first-time clients of software contractors, or first-time managers of software developers. "Just make it match these mockups!" Dragging a little Facebook icon into place and poof all the code and edge case handling and database work needed to integrate with Facebook is all done, put to bed, etc. Just put that icon in the right place.

kamaal 15y ago
Well in such a case those people should not opt to manage technical people.

robomartin 15y ago

P = programmer; NP = non-programmer NP: How long will it take? P: It'll take as long as it takes. NP: I don't understand. What does that mean? P: When you tell me that the product is finished, it's done. NP: You mean you can't estimate how long it'll take? P: I can't estimate when you'll decide that it is done, therefore an accurate prediction is impossible. From experience I know that a project like this can take from three to nine months. A lot depends on you, the choices you make and how well you communicate them to me. A lot also depends on how many times you change your mind during the process and how many times you change the specifications during the job. Major changes can even cause the project to be scrapped and start over. Rest assured that I will not be the one slowing this down. NP: How much will it cost? P: It will cost as much as it costs. NP: I don't understand. What does that mean? P: You just learned some programming! Go back to your first question and re-read the entire thing. This is called an endless loop.

bravura 15y ago
What are the implications on the universe if P=NP?
- deleted 15y ago
  [deleted]
buu700 15y ago
The real question is, are they the same person?
doktrin 15y ago
This takes a fairly narrow view of a development process, IMO. The "but you didn't specifically tell me to do X" attitude can border on the irresponsible and/or lazy. It's all nice and well to expect accurate step-by instructions to be communicated by a PM, but the onus also does fall on developers to meet them half-way. While machines may require explicit and literal step-by instructions, a human being value-add is in thinking rationally, making inferences and (when necessary) asking questions.
15y ago

robomartin 15y ago

One of my favorite ones was a fellow who wanted an iPhone and iPad game done. He proclaimed that it had to be done for $10K. This game easily required six months of work. It also required server-side development and support. He also had no idea that every so often you need to fix, I mean, update, your apps because new iOS releases might break them. I was open-minded and decided to invest some time educating him on the process and the needs of the platform. Once he had enough information he networked and hired someone out of India to build him the game. I was not happy about that at all. Then I learned that he was having all kinds of problems with the process (and the app) being a total mess. He's learning his lesson.

yardie 15y ago
Come on now. Don't blame the programmer being Indian for the problems your friend is having. Good programmers anywhere cost almost as much as they do in the US now. Globalization means that an Indian programmer has the same access to salary information in the west. He/she might knock off 10% to be competitive. You're friend could have easily got a cheap, inexperienced programmer anywhere else in the world, including the US, and still had the same problems. He had a budget, you knew it wasn't enough for what he needed, and he came up snake eyes.
- dpritchett 15y ago
  If an unsophisticated client looks to another hemisphere to get a one-off iOS app built I wouldn't bet "world-class talent at a 10% discount" was their hiring strategy. More likely "I can get this done for 1/3 of what this guy is quoting me? What could possibly go wrong?"
- robomartin 15y ago
  Who says I am blaming the programmer for being Indian? I'm not. Excellent work being done out of India. Here's the problem, lots of people like this fellow look at India and China to find the lowest possible bidder out of complete ignorance. And, when they find the lowest possible bidder they get exactly that: crappy inexperienced programmers. If they searched for the lowest possible bidder in the US or Europe they'd get exactly the same thing. I did not intend to imply that Indian programmers are not good. An ignorant fool looking for rock bottom prices will, more than likely, find them in India rather than the US or Europe.

b1daly 15y ago

I'm a music producer, and this is a constant problem for me that after many years still boggles my mind. The root of the problem is that producing a modern style recording of music is resource expensive. Far more than anybody would like it to be. The resources can be time/money/talent. So there is an intense bias on everyone involved in the project to under-estimate. I usually work on low budget projects. You can actually accomplish a lot with a low budget. But as you start to climb the "quality" scale the costs increase much more quickly than the quality. To get a product that you might consider subjectively twice as good might cost ten times as much. At core, I feel like there is a fundamental principal of information organization. It takes a certain amount of resources to undo entropy and organize information. When we experience a piece of work we can intuitively feel that entropy has been significantly reduced, we appreciate the accomplishment. (Even if we are not in the field of endeavor we still feel it though we don't know how it is done.) One specific point of difficulty in estimating time in music production is that there is an x-factor. It derives from the need to create something exceptional, which has both subjective and objective elements. With an experienced production team you can predict how long it will take to produce something competent. But usually there is a point where we have all done the things we know how to do: the song is arranged, the players are good, the production environment is good, the instruments are good. But the song is missing something. What is it? It is not known yet. So an iterative trial and error process begins. Let's try a different singer. Maybe we should borrow my friends guitar. Let's keep recording. Let's rewrite the song. Hopefully a solution is arrived at, but since it is a trial and error process it is impossible to estimate how long it will take.

jtheory 15y ago

Is Pachelbel's Canon considered complicated as far as music performance goes? Reading that knocked me right out of the flow of this article. What's he thinking of? Maybe someone's complicated improvisations on top of the (really simple) canon? On topic: the post didn't really cover where the layperson's base estimate comes from -- a layperson sees X pages, or X features, and multiplies that by... what? The reference they use varies hugely, in my experience; many are trying to imagine what's involved in developing a "feature" -- which yes, will generally be a massive underestimate... not always, though. There are also plenty of laypeople who imagine that everything they see is incredibly complex & difficult to create (I have a neighbor who imagined that writing software meant typing in the gunk he saw when accidentally opening a binary file in notepad), and all written from scratch -- so a simple blog would take ages to build, because it's obviously complicated stuff. In real life, some quite complicated things come for free, other seemingly simple things take gobs of time, sometimes unexpectedly... it's a big mess. And just as some of the harshest wildcards (like nasty bugs in old browsers) are calming down, more form factors, interaction modes, etc. are popping up, so I'm not sure it's going to be a problem we can stop talking about anytime soon.

damncabbage 15y ago
Is Pachelbel's Canon considered complicated as far as music performance goes? Consider the more complicated violin part.
- jtheory 15y ago
  Still, you hear grade schoolers play it. And wow, if you've ever been forced to play the part of anything other than violin -- I played the viola part innumerable times in HS orchestra, and the cello part was even worse -- it's almost a punishment.

NameNickHN 15y ago

Why does the way we naturally measure complexity stop working when we apply it to programming? Lots of projects that have nothing to do with programming end up taking longer than expected or are over budget.

jsvaughan 15y ago

"But it brings up another more interesting question: why does the way we naturally measure complexity stop working when we apply it to programming?" Ah and I had high hopes for that next paragraph, There are two fundamental differences between, lets say, estimating how long it will take to make a sandwich, and estimating how long it will take to create an online noticeboard for your village. The first difference is that one task is manufacture, the second task is design. Writing code IS design, and it is inherently much harder (or even logically impossible) to estimate because implicitly you do not know what done means. The second difference is that (broadly) how satisfactory a sandwich or other real world object is, is proportional to how long it takes to make. This is not true for software, because it is not a physical entity. It can be tripped up by a single incorrect character, and then, forget tasty - your software sandwich doesn't even exist.

scanr 15y ago

Any project management software out there that takes an ML / probabilistic approach to this? One thing I found at one of the mega-corps that I worked at is that both project managers and regular managers were blind to history. Note: this was before agile became accepted by mega-corps. Manager: How long is this going to take? Me: Virtually impossible to predict accurately Manager: I need a date Me: 3 months Manager: That's too long, we need to get it done in 1 month Me: OK, 1 month ... 5 months and several iterations later ... Manager: Project has been deprioritized, we have a new project, how long is this going to take? Me: Virtually impossible to predict accurately Manager: I need a date Me: I need you to be stateful.

Scotchy 15y ago

I very often hear this comment from my fellow developper colleagues. In our startup, I am responsible for the frontend coding. Very frequently, I get requests from other developpers to implement this or this, and very often those requests are annotated by a "c'mon, it is just CSS, it'll only take you 5 minutes" when they have absolutely no idea what they are talking about, because they never ever did real frontend developpment. They simply think that everything boils down to HTML and that frontend is NOT really programming. Attempting to explain that a frontend also has logic similar to backend, and that no, plotting graphs when you don't even have the data modeled is not just a 2 liner with Google Visualisation. I am always offended by people estimating time for work they never did. So I find myself very rarely giving estimations. Sometimes when I have a very precise idea on how to implement an idea right when I'm being told about it, I usually say "yeah this is definitely doable in a short time span". Meaning things that do not take more than half a day to implement, because you already compiled the code in your brain. Now, when a colleague of mine estimates stuff for me, I tell them "No I don't think this is going to take X minutes to implement. If you think it is implementable so fast, why don't you go ahead and do it.". When they take it to the word, and I see them still struggling after many hours on some problem that are trivial to me, I will go ahead and help them out. After a few times, they usually won't bother me with estimations anymore... NOT.

adrianhoward 15y ago

I think it's much more domain-expert vs non-domain-expert rather than programmer vs non-programmer. I've regularly seen very experienced developers make exactly the same sort of mistakes when talking to people outside of their field (Sales, Design, etc.) because they miss things like: * Sales is about building stories and relationships - it's not just listing features and meeting people. * Design is about understanding the user, iterating, testing and experimentation - it's not just making things pretty. I wish technical folk were naturally better at this sort of thing - but they (and I :-) keep making similar sorts of mistakes. (It's also amusing to see the "it's not like building a bridge" analogy turn up. Go talk to some engineers, architects and builders - and find out how many of those projects ended up matching the original time and cost estimates :-)

DanBC 15y ago
> (It's also amusing to see the "it's not like building a bridge" analogy turn up. Go talk to some engineers, architects and builders - and find out how many of those projects ended up matching the original time and cost estimates :-) That's an important point. I used to work in sub-contract electronic engineering. We had a lot of information about how long it took to do things, and how much items cost. In theory, estimating the cost of projects should have been straightforward. And then you add 15% for wiggle room. And still it's hard to keep things going out on time and on budget.

Heisens 15y ago

Such a noob topic. If you don't know what you are doing, just admit it. For seasoned programmers and PMs, this is a non-issue. Perhaps you are in the wrong profession if you can't handle the demands without whining like a brat.