6 ms·
There are 2 reasons: 1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It i
by rdtsc 11y ago
There are 2 reasons:
1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice.
2) Break down in communication between management (product managers) and engineers who are writing code.
On the first part. Usually junior software developers simply have not had experience shipping software in a full software cycle. From initial requirements, to design, to deployment, to maintenance. They did projects perhaps in school and homework exercises. Later some eventually learn that they chronically underestimate the time, and just start adding a factor (2x) to what their initial estimate is. That is a step up. But many never learn. There is also a race to the bottom -- the ones that promise shorter timelines get promoted and seen as smarter, more aggressive, better team players. The ones that are realistic are seen as maybe "taking it easy", "not working hard enough". Then everyone sees who gets promoted and show doesn't and learn -- "Aha I need to be like cool cowboy coder Joe over there, he is now running his own department". And so it spreads. Notice how there is often no feedback loop to punish the cool cowboy coder after the project is not delivered on time.
On the second part -- even if there are engineers who know how to estimate the time, nobody asks them. Because the person who promised the feature already played golf with the customer and told them they'll get the new shiny toy in 6 months. That person might have never been a developer, they might have an MBA or where a football coach or something like that. And they usually have a higher position in the company. So they promise and then everyone is pushed to deliver on that deadline. If it fails the failure is usually attributed to the engineers doing the work because they haven't worked "hard enough".
- exelius 11y ago> 1) Developers cannot reliably estimate software timelines. That often comes with experience and age, but many never learn that skill. It is a skill like programming itself. It comes with practice. This is true. But this is the problem that Agile is supposed to help solve -- with the caveat that most companies don't do true Agile. I guess people are now calling it Continuous Delivery, but the idea is the same: you don't tell people when a feature will be done, only when you will be working on it. But even then, even with experienced estimators, the rule of thumb is to add 15-30% to their estimates. > 2) Break down in communication between management (product managers) and engineers who are writing code. Totally agree that this is very common. Unfortunately, the best way to solve this is with process, and developers hate process. But developers often don't realize that they can use the process to their advantage: a strong engineering lead can refuse to accept poorly defined requirements. That allows you to push back on product to force them to define the requirement better. IMO this breakdown is caused by just that: developers who say "oh yeah, I can do that", then start working on the problem and realize that there are a lot of undefined edge cases. If they promised it to the client in 6 months, fine, but define what "it" is. From the perspective of an MBA, if we have to bring in a contractor to implement it, fine. I know where I can go to hire certified rockstar contract developers at $200/hr, which I will do if the client is lucrative enough. There is no feature I can think of that would take more than 6 months of development.
- moron4hire 11y agoProcess doesn't work, it can't work, because Process isn't the problem, it's top-down management infrastructure. Process is orthogonal to that. In a top-down management style, your so-called "Lead Developer" is just the first head on the chopping block. They're leaders of puppet regimes. If the real leaders don't like them, they install a new one. It comes in many forms: layoffs, "performance" related dismissal ("you're not getting enough done!" because of all the rejected requirements docs), even reassignment to "re-engineering initiatives" that have no real chance of seeing the light of day. Finding a management infrastructure that actually cares about feedback is incredibly rare. I have personally never found it, in 10 years across 6 companies. No amount of process is going to change a bad manager.
- exelius 11y agoAny company of a certain size must convert to a top-down management infrastructure. I've seen what happens in large companies that don't -- you get 15 teams building different versions of the same product, and internal politics decide whose version becomes the 'official' one. But invariably, the internal user community settles on the one that's actually best for their use case, so you end up having to support 3 or 4 versions of every product you create. It's an absolute support and integration nightmare. You talk about a business like it's supposed to be a democracy. It's not. You need clear direction, and democracy is messy and opaque if your company is too big for you to know the name of everyone who works there. Now the way you describe top down management is a bad implementation of it -- executives need to work to develop their middle managers -- but every company eventually has to convert from a democratic decision making process to a chain of command. I've been working recently with a number of household-name startups that are having this exact issue. They've hit a plateau in their growth because the culture is dead-set against top-down management, but the company is now too big for people to self-organize. Engineering initiatives that would have taken 2 months in startup mode now take 2 years because they have to go back and rebuild half their systems to make them flexible enough to support the multiple use cases that they now have. Or because they were a startup, they didn't worry about geographic redundancy because that wasn't part of the MVP. So it was put off for later, and now they're a billion dollar company operating out of a single datacenter because their databases and applications were built without multi-datacenter being a requirement. Product can't even reliably tell engineering what features to implement because nobody on the product side has enough authority to shout over the crowd. So having an infrastructure that cares about feedback is good, but the infrastructure also needs to empower people to do something about said feedback.
- DougWebb 11y agoI've spent a lot of time thinking about why software projects go over budget so often. I think that developer estimation only plays a minor role, and the big problem is failure to manage project scope. I'm specifically talking about projects which go like this: 1) A set of high-level business requirements are put together, and given to a tech lead to estimate. 2) The tech lead's estimate, based on the requirements as-given, is used to determine a budget for the project, and probably a schedule as well. 3) Once the budget and schedule are approved, the project commences. 4) As development begins detailed software requirements are worked out, which usually exposes holes in the high-level business requirements that need to be fleshed out as well. 5) The estimates are never updated, and any attempt to increase the budget due to the new requirements are seen as a sign of failure. What I see happening in #4 is two kinds of scope creep. The first kind is the sort everyone is familiar with: a requirement was missed, a new feature needs to be added, and since this wasn't in the original budget the budget needs to be increased. It's easy to identify these new/changed requirements, easy to see the big chunk of time needed to implement them, and easy to make prioritization decisions about the new requirement vs the other original requirements, so that maybe something can be dropped in order to avoid increasing the budget. But there's a second kind of scope creep, and it's insidious. It comes from turning high-level requirements into detailed requirements into implemented and tested code. Every decision a developer needs to make about how to implement a feature is an implicit detailed software requirement, and you just can't know how many of those decisions will be needed or what their impacts will be until you get into the details. The time needed for these all add up, and I consider them to be scope creep because they aren't part of the high-level requirements used for the estimate and budget. We try to add padding for them because we know they're going to happen, but if you add sufficient padding across-the-board for all of the detailed requirement decisions that might be needed your project will never get approved. So we pad insufficiently, hoping that some of the requirements will be easier to implement than we estimate to cover the ones that aren't, and we don't manage this kind of scope creep at all. It's invisible, because it's just adding a few minutes here, a few minutes there, until our estimates have been blown away and we're over budget and we can't really say why.
- rdtsc 11y agoI agree with you. I was mostly thinking of one iteration (a simplistic view) of "we know the scope, we go to work, we deliver (or fail) the result" which happens only in small projects. The two types of scope creep is a good observation, I'll have to remember that idea.
- sopooneo 11y agoYour point about the overly aggressive golf-made deal: how do we fix that? I really don't have an answer, but if we, as developers, are really so smart, we should be able formulate the problem and find a solution. We will have to avoid imagined constraints and thinking the answer is related to producing more code, but it's got to be there. Is it related to playing the office politics game better? And if so, can that be a learned and tolerable skill? Is it to do with organizing? With preferring self employment or contracting? I am probably naive, but I think if we say: the situation is now X and I want to be Y, we can find a way to get there.
- rdtsc 11y agoCompany I work for tried to solve it using this rule -- don't hire any MBAs. Every manager, product manager, and even salesperson is (or at least was at some point) an engineer. This works in our context (and has worked for many decades apparently). But I doubt this scales to a large size.
- avn2109 11y agoIIRC Palantir subscribes at least nominally to this philosophy, too. And they're trying to scale it, so we'll get to see how that works out in the near-ish future. (Unless you work at Palantir, in which case I feel silly).
- ryandrake 11y agoThat sounds like a very silly policy. Would your mind be blown to learn out there exist many MBAs with engineering backgrounds, and that they're actually not idiots?
- rdtsc 11y agoIt is silly I agree but it worked for many decades, in the sense the customers are happy and engineers here seem to be happy. One can also say it didn't work because the company hasn't turned into Google or "existed" with a huge cash windfall for founders. Of course that was just one variable and it wasn't isolated. There could have been other factors and reasons for the said result and it might not have had anything to do with "no MBAs" rules. I just presented it as a reference or what others do as a "solution".
- xivzgrev 11y agoGood points but there are also company wide reasons, namely self absorption among leadership. For example, fear. Fear of running out of money, fear of losing to competition, fear of tarnishing brand if we don't respond to this service issue right away, fear of insignificance, etc. impatience - we got a really promising thing and I want to fucking make it/make my mark/cash out! Greed - if we squeeze everyone a little harder I can make more money (even though I'm really well off). These are all real human struggles. I have them for sure. I just wish you find a place with self aware leadership that endeavors to keeps these emotions in check. Usually startups with leaders who have families are a little better because their family essentially forces them to focus on someone else, at least a little.
- libria 11y ago> 1) Developers cannot reliably estimate software timelines. Disagree. Developers can provide reliable low risk estimates that will most certainly be rejected or truncated by management. The problem is not estimates, the problem is every manager fantasizing that their Captain Kirk strategy, "I'll have it fixed in 4 minutes, Captain" "Do it in 2." is some form of motivating leadership.
- eclipxe 11y agoEVERY manager? Hyperbole there...
- libria 11y agoI misspoke. I meant more along the lines of > the problem is any manager If your greater point is that some developers really are poor estimators, I agree, but all of them operate under pressure to err on the optimistic side.
- dasil003 11y agoThere are two ways to parse that sentence: 1. Every manager does that 2. Every manager that does that is the problem
- JohnBooty 11y agoI agree with what you said. I think there's also a third primary reason and maybe a fourth: 3) Developers are expensive. If you have a team of 5 developers each being paid a salary+benefits of $1,500-$3,000 per week and they tell you a feature will take four weeks, management does a little mental math and says holy shit, that's a lot of money and it's understandably tempting for them to want to see if that team really needs four weeks, or if maybe it can be done in three weeks. (Meanwhile, as you said, the feature will probably actually take six weeks because developers cannot reliably estimate...) 4) Developers have a hard time saying "no." We have a culture of working crazy hours. Management knows this and often intentionally or unintentionally exploits it.