9 ms·

Why Software Projects Are Terrible And How Not To Fix Them

136 points by drewcrawford 15y ago

nhashem 15y ago
A couple years ago, when I was at a previous employer, I was lamenting the typical corporate frustrations to a friend. The executives seemed to have a new "strategy" every quarter. All our projects had unrealistically given deadlines. Even when we successfully released a new product, it was basically DOA, because they were never things we would use ourselves. It was basically impossible to pivot our aging flagship product, because it has grown so complex that it was impossible to get any more incremental gains without a huge commitment in resources. At least once a month I'd be asked, "Well can we make this change to the product quickly?" and I would always say, "No. There are no quick changes anymore. You can either commit the staff to getting the product back into a state where we can do quick changes again (ie. refactoring some of it), or you can just put the product in maintenance mode and have the staff do something else." And yet no matter how many times I explained why, using as many MBA-friendly concepts and analogies as possible, they never understood. My friend asked if any of the executives and managers had an engineering background, and I literally laughed out loud. Of course they didn't. If they did, they would have understood concepts in this article. Instead they were exactly like that political douchebag in The Wire video. One of my greatest hopes for the current proliferation of startups is that some of the technical talent there actually does end up as leaders corporate America. I see the fundamental problem of corporations led by execs with the classic MBA-educated, McKinsey/Bain-trained background is they've never created or produced a damn thing in their lives, and just fundamentally don't understand why their decisions are so dysfunctional.
- sixbrx 15y ago
  If not understanding you is the worst that happens I'd say you're lucky, but watch out... My buddy and I have been resisting "quick changes" on projects that have a lot of technical debt for a number of years, and what has happened is that managers have just gone to some other less experienced developers who have no scruples to get their changes done without the "fuss". Our reputations have taken a bit of a hit as well - we're "too careful" or don't see that the "perfect is the enemy of the good" etc. That's what they say in front of us, I'm sure they're more colorful when we aren't listening. I've learned to mitigate this by working with a couple of managers that aren't clueless but it's a danger when that isn't an option...
  - tawman 15y ago
    Technical debt is a major red flag especially in a savvy organization that realizes it's accumulating it. Especially if much of the early success relied on it. I was talking with a prospective client that had spent the past three years rushing a SaaS solution to market. They even referenced the technical debt as an area they need help with overcoming as they move forward. Second order change is difficult for any organization or established project, but remaining consistent with your message can drive the process in that direction. Sounds like a challenge for the two of you, but try to bring solutions / recommendations instead of minor resistance that might get misinterpreted. Not easy I know.
    - unobfuscate 15y ago
      How do you spend 3 years rushing a product to market? What type of system were they building?
      - tawman 15y ago
        This company operates in the e-medical records space and works with hospitals, physicians, and other providers. It is more than just records management and deals with referrals for service along with analytical capabilities etc. Hard space to penetrated but with a lot of federally funded initiatives the past few years. New regulations around EHR is driving the space.
  - pasbesoin 15y ago
    I've reached the point where I refuse to enable such people, even if it means short term sacrifice for myself. They need to die off, in a career sense. Any help I give them, even when well qualified and executed on my part (i.e. despite them / their attitude) is just adding to the problem. Seriously. There are a lot of "morons" and "jerks" out there who need, simply if painfully, to be "disenabled". (And, to some extent, if their/the customers suffer, well, those people voted with their feet in becoming customers. In our "free market" society (this is a U.S./"Western" perspective), it's somewhat akin to our politics: You can bitch all you want about the politicians and how they are screwing you over, but YOU elected them.)
Natsu 15y ago
People who have too many problems dump them onto someone else. Such people in management make life miserable for anyone below them. Problems tend to move around in circles and pointing out an issue is a good way to have it boomerang back to where it becomes yours to solve. It's much better if you have coworkers and especially management with whom you can cooperate. If there's at least enough flexibility to improve things, it's possible to have things snowball in reverse, as it were, where each person tackles a piece of the problem and it gets taken care of. But it's hard if you work with people who don't care. For example, it can be frustrating to get blamed for causing problems that have always existed, with no one noticing them, simply because people were too far away from the consequences and those who suffered the consequences had no idea about their cause. And yes, even simple things like "source control" can be a tough sell when the devs insist that keeping the .c file in the folder with the .exe counts as source control. For a nearly 30-year-old DOS program, written with ASCII menus (think printf, not ncurses), uses only statically assigned globals (malloc? what's that do?), deeeeply nested ifs, that directly writes outputs to hardware I'm not sure anyone at the company still understands, and there's a custom build for each install? I'm just grateful I have nothing to do with "maintaining" that source and that DOS makes it easy to do backups via xcopy and a network drive. In retrospect, I might have been happier not knowing what was in those .c files.
pagekalisedown 15y ago
Original source of the dilbert comic in the article: http://dilbert.com/strips/comic/2006-11-05/ http://dilbert.com/strips/comic/2006-11-05/ dilbert can be painfully accurate, to the point where you have to stop reading it to avoid depression.
yason 15y ago
You can't really fix things like software publicly. That'd never fly in a typical company, startups excluded of course. The only thing you can do is do it secretly and there are practically two ways to accomplish that. The first way is hard to do if there's a large group of programmers working on the project but it's very suitable for older projects that is still under development but with lower staffing. If possible, do things efficiently locally (use source control, use python/jython, generate code, do automated builds, do automated testing, reuse, and only deliver what you must) but leave your advantageous practices unadvertised. Then, slowly abuse your edge to steal maintenance and refactoring time from actual to-be-done tasks and features. Eventually you can ease your own future work and you'll get more time to work on the system instead of the features which become easier to add. The second way is to employ a targeted attack against a whole malfunctional group or team at once: skunkworks projects. Do the damn thing from the scratch and do it right, then replace the old steaming pile of junk with it before anyone gets to say no, and weasel your way back to the surface by toting things were just changed a bit, and never replaced. Of course, this comes with the assumption that the evil is in the system and your colleagues aren't actively demolishing everything you build.
- ido 15y ago
  Do the damn thing from the scratch and do it right, then replace the old steaming pile of junk with it This reminds me of the many attempts to rewrite nethack in a higher level language, like ruby or python. They all fail miserably, leaving nothing but a smoking crater and piles of code that is never to be used. Turns out all those millions of lines of c code written over a period of 25 years actually do quite a lot.
  - morsch 15y ago
    Yes. Programmers (myself included) constantly want to rewrite form scratch: But this time, we're going to do it right! Unfortunately, often, you'll draw exaggerated conclusions from past failures and over-engineer things ("All problems in computer science can be solved by another level of indirection."). You might also end up creating new design problems, e.g. a bottleneck in an area that the old design just happened to avoid. And even if you do get everything right design-wise, you'll always end up creating new bugs along the way. (Obviously, none of this is a new insight, the pros and cons of rewrite from scratch have been discussed back and forth forever.)
gregschlom 15y ago
>Turning around a software project is like turning around a tugboat: it takes a lot of time and energy, and tremendous momentum. Off-topic sailors nitpick: tugboats are highly maneuverable, what takes a lof of time and energy is turning around a very large container ship with a tugboat See for example: http://www.youtube.com/watch?v=rkCqDLR2s78 http://www.youtube.com/watch?v=rkCqDLR2s78
- jimfl 15y ago
  Indeed few boats that size are as maneuverable as a tug. It's a good metaphor for a skilled developer: even with that kind of agility, it can take a while to change the direction of something with a huge momentum. And a software project has momentum vectors shooting off in high dimensional space.
cateye 15y ago
There is a strong correlation between terrible projects and the value that the project will deliver. The lower the future value and the uncertainty that it will bring the value, the more problems the project will experience. So, if you as a developer ask "how is this going to deliver value (make money)?" and you don't get a convincing and logical answer the more problems you will experience. The easy part is, the bad projects are really easy to identify because no one on the project can tell something argumentative. - "You have to build a video website in 1 week!" * How is this going to make money? - We will charge every user for every video they upload * So, it is like Youtube but paid. Why will people use this instead? - BSOD...
alphadogg 15y ago
IMO, one thing this article seems to paint inadvertently and incorrectly is that the problem, if there is any, (likely) lies entirely at the feet of management. It takes two to tango. The article isn't wrong, but it's only the elephant's leg that the blind man is looking at.
mattmanser 15y ago
Has this guy ever even read Joel's stuff? The acronym TDD keeps popping up time and again. Joel's not a TDD fan last I heard him talking about it (admittedly stackoverflow podcasts a couple of years back). He certainly wasn't a fan of it when the Joel test came out because that pre-dated TDD. Fact is, while I agree with some of the points in the article, there are things that aren't accepted practice and TDD is one of them. Personally I think it's a colossal waste of time apart from some complicated methods. And don't even get me started on the esoteric 'scrum' or 'agile' where everyone has their own pet definition of what it really means.
- yannickt 15y ago
  He never said or implied that Joel is a TDD fan in the article. He also concedes that TDD might be a controversial practice in the first paragraph. In any case, the merits of any particular practice have little to do with the main thesis of this blog post.
wundie 15y ago
UX comment on this website: I like to click and highlight words as I read. I know I'm not the only one who does this. On this site when I do this, the article keeps drawing focus to the top of the page and its...annoying. Why would someone do this? I've never experienced this issue on any other site before.
- civilian 15y ago
  I thought I was alone in that preference! A designer was watching me read through an article and he was like ".... why are you clicking everywhere?" It helps me keep track of my place, and I'm ready to control-copy even faster.

136 points by drewcrawford 15y ago

A couple years ago, when I was at a previous employer, I was lamenting the typical corporate frustrations to a friend. The executives seemed to have a new "strategy" every quarter. All our projects had unrealistically given deadlines. Even when we successfully released a new product, it was basically DOA, because they were never things we would use ourselves. It was basically impossible to pivot our aging flagship product, because it has grown so complex that it was impossible to get any more incremental gains without a huge commitment in resources. At least once a month I'd be asked, "Well can we make this change to the product quickly?" and I would always say, "No. There are no quick changes anymore. You can either commit the staff to getting the product back into a state where we can do quick changes again (ie. refactoring some of it), or you can just put the product in maintenance mode and have the staff do something else." And yet no matter how many times I explained why, using as many MBA-friendly concepts and analogies as possible, they never understood. My friend asked if any of the executives and managers had an engineering background, and I literally laughed out loud. Of course they didn't. If they did, they would have understood concepts in this article. Instead they were exactly like that political douchebag in The Wire video. One of my greatest hopes for the current proliferation of startups is that some of the technical talent there actually does end up as leaders corporate America. I see the fundamental problem of corporations led by execs with the classic MBA-educated, McKinsey/Bain-trained background is they've never created or produced a damn thing in their lives, and just fundamentally don't understand why their decisions are so dysfunctional.

sixbrx 15y ago
If not understanding you is the worst that happens I'd say you're lucky, but watch out... My buddy and I have been resisting "quick changes" on projects that have a lot of technical debt for a number of years, and what has happened is that managers have just gone to some other less experienced developers who have no scruples to get their changes done without the "fuss". Our reputations have taken a bit of a hit as well - we're "too careful" or don't see that the "perfect is the enemy of the good" etc. That's what they say in front of us, I'm sure they're more colorful when we aren't listening. I've learned to mitigate this by working with a couple of managers that aren't clueless but it's a danger when that isn't an option...
- tawman 15y ago
  Technical debt is a major red flag especially in a savvy organization that realizes it's accumulating it. Especially if much of the early success relied on it. I was talking with a prospective client that had spent the past three years rushing a SaaS solution to market. They even referenced the technical debt as an area they need help with overcoming as they move forward. Second order change is difficult for any organization or established project, but remaining consistent with your message can drive the process in that direction. Sounds like a challenge for the two of you, but try to bring solutions / recommendations instead of minor resistance that might get misinterpreted. Not easy I know.
  - unobfuscate 15y ago
    How do you spend 3 years rushing a product to market? What type of system were they building?
    - tawman 15y ago
      This company operates in the e-medical records space and works with hospitals, physicians, and other providers. It is more than just records management and deals with referrals for service along with analytical capabilities etc. Hard space to penetrated but with a lot of federally funded initiatives the past few years. New regulations around EHR is driving the space.
- pasbesoin 15y ago
  I've reached the point where I refuse to enable such people, even if it means short term sacrifice for myself. They need to die off, in a career sense. Any help I give them, even when well qualified and executed on my part (i.e. despite them / their attitude) is just adding to the problem. Seriously. There are a lot of "morons" and "jerks" out there who need, simply if painfully, to be "disenabled". (And, to some extent, if their/the customers suffer, well, those people voted with their feet in becoming customers. In our "free market" society (this is a U.S./"Western" perspective), it's somewhat akin to our politics: You can bitch all you want about the politicians and how they are screwing you over, but YOU elected them.)

Natsu 15y ago

People who have too many problems dump them onto someone else. Such people in management make life miserable for anyone below them. Problems tend to move around in circles and pointing out an issue is a good way to have it boomerang back to where it becomes yours to solve. It's much better if you have coworkers and especially management with whom you can cooperate. If there's at least enough flexibility to improve things, it's possible to have things snowball in reverse, as it were, where each person tackles a piece of the problem and it gets taken care of. But it's hard if you work with people who don't care. For example, it can be frustrating to get blamed for causing problems that have always existed, with no one noticing them, simply because people were too far away from the consequences and those who suffered the consequences had no idea about their cause. And yes, even simple things like "source control" can be a tough sell when the devs insist that keeping the .c file in the folder with the .exe counts as source control. For a nearly 30-year-old DOS program, written with ASCII menus (think printf, not ncurses), uses only statically assigned globals (malloc? what's that do?), deeeeply nested ifs, that directly writes outputs to hardware I'm not sure anyone at the company still understands, and there's a custom build for each install? I'm just grateful I have nothing to do with "maintaining" that source and that DOS makes it easy to do backups via xcopy and a network drive. In retrospect, I might have been happier not knowing what was in those .c files.

pagekalisedown 15y ago

Original source of the dilbert comic in the article: http://dilbert.com/strips/comic/2006-11-05/ http://dilbert.com/strips/comic/2006-11-05/ dilbert can be painfully accurate, to the point where you have to stop reading it to avoid depression.

yason 15y ago

You can't really fix things like software publicly. That'd never fly in a typical company, startups excluded of course. The only thing you can do is do it secretly and there are practically two ways to accomplish that. The first way is hard to do if there's a large group of programmers working on the project but it's very suitable for older projects that is still under development but with lower staffing. If possible, do things efficiently locally (use source control, use python/jython, generate code, do automated builds, do automated testing, reuse, and only deliver what you must) but leave your advantageous practices unadvertised. Then, slowly abuse your edge to steal maintenance and refactoring time from actual to-be-done tasks and features. Eventually you can ease your own future work and you'll get more time to work on the system instead of the features which become easier to add. The second way is to employ a targeted attack against a whole malfunctional group or team at once: skunkworks projects. Do the damn thing from the scratch and do it right, then replace the old steaming pile of junk with it before anyone gets to say no, and weasel your way back to the surface by toting things were just changed a bit, and never replaced. Of course, this comes with the assumption that the evil is in the system and your colleagues aren't actively demolishing everything you build.

ido 15y ago
Do the damn thing from the scratch and do it right, then replace the old steaming pile of junk with it This reminds me of the many attempts to rewrite nethack in a higher level language, like ruby or python. They all fail miserably, leaving nothing but a smoking crater and piles of code that is never to be used. Turns out all those millions of lines of c code written over a period of 25 years actually do quite a lot.
- morsch 15y ago
  Yes. Programmers (myself included) constantly want to rewrite form scratch: But this time, we're going to do it right! Unfortunately, often, you'll draw exaggerated conclusions from past failures and over-engineer things ("All problems in computer science can be solved by another level of indirection."). You might also end up creating new design problems, e.g. a bottleneck in an area that the old design just happened to avoid. And even if you do get everything right design-wise, you'll always end up creating new bugs along the way. (Obviously, none of this is a new insight, the pros and cons of rewrite from scratch have been discussed back and forth forever.)

gregschlom 15y ago

>Turning around a software project is like turning around a tugboat: it takes a lot of time and energy, and tremendous momentum. Off-topic sailors nitpick: tugboats are highly maneuverable, what takes a lof of time and energy is turning around a very large container ship with a tugboat See for example: http://www.youtube.com/watch?v=rkCqDLR2s78 http://www.youtube.com/watch?v=rkCqDLR2s78

jimfl 15y ago
Indeed few boats that size are as maneuverable as a tug. It's a good metaphor for a skilled developer: even with that kind of agility, it can take a while to change the direction of something with a huge momentum. And a software project has momentum vectors shooting off in high dimensional space.

cateye 15y ago

There is a strong correlation between terrible projects and the value that the project will deliver. The lower the future value and the uncertainty that it will bring the value, the more problems the project will experience. So, if you as a developer ask "how is this going to deliver value (make money)?" and you don't get a convincing and logical answer the more problems you will experience. The easy part is, the bad projects are really easy to identify because no one on the project can tell something argumentative. - "You have to build a video website in 1 week!" * How is this going to make money? - We will charge every user for every video they upload * So, it is like Youtube but paid. Why will people use this instead? - BSOD...

alphadogg 15y ago

IMO, one thing this article seems to paint inadvertently and incorrectly is that the problem, if there is any, (likely) lies entirely at the feet of management. It takes two to tango. The article isn't wrong, but it's only the elephant's leg that the blind man is looking at.

mattmanser 15y ago

Has this guy ever even read Joel's stuff? The acronym TDD keeps popping up time and again. Joel's not a TDD fan last I heard him talking about it (admittedly stackoverflow podcasts a couple of years back). He certainly wasn't a fan of it when the Joel test came out because that pre-dated TDD. Fact is, while I agree with some of the points in the article, there are things that aren't accepted practice and TDD is one of them. Personally I think it's a colossal waste of time apart from some complicated methods. And don't even get me started on the esoteric 'scrum' or 'agile' where everyone has their own pet definition of what it really means.

yannickt 15y ago
He never said or implied that Joel is a TDD fan in the article. He also concedes that TDD might be a controversial practice in the first paragraph. In any case, the merits of any particular practice have little to do with the main thesis of this blog post.

wundie 15y ago

UX comment on this website: I like to click and highlight words as I read. I know I'm not the only one who does this. On this site when I do this, the article keeps drawing focus to the top of the page and its...annoying. Why would someone do this? I've never experienced this issue on any other site before.

civilian 15y ago
I thought I was alone in that preference! A designer was watching me read through an article and he was like ".... why are you clicking everywhere?" It helps me keep track of my place, and I'm ready to control-copy even faster.