21 ms·

Symptoms of Dysfunction in Software Teams

103 points by buzzkills 12y ago

groovy2shoes 12y ago
I've been stuck maintaining a carcinogenic prototype for the last 2 months. While I don't like dealing with the maintenance nightmare, I do really like that term. I'd never heard it before, but I knew what it was as soon as I read it -- the text in that section only confirmed my suspicion.
- knieveltech 12y ago
  Yeah, this definitely feels like a permanent addition to the lexicon. It crisply describes at least two systems I'm maintaining currently.
  - wmeredith 12y ago
    If it gets canonized, I'd much prefer "cancerous prototype" because it's easier to say.
    - MaysonL 12y ago
      "metastatic prototype"
      - cpr 12y ago
        You mean metatastic prototype?
    - schnevets 12y ago
      I like this better too - the prototype will inevitably grow from its humble origins, and with every new feature that is hastily thrown in, the application becomes ever so much riskier.
    - jmcphers 12y ago
      Also known as: Quick Hack, Kleenex Code, Disposable Code, Scripting, Killer Demo, Permanent Prototype, Boomtown. [1] http://blog.codinghorror.com/the-big-ball-of-mud-and-other-architectural-disasters/ http://blog.codinghorror.com/the-big-ball-of-mud-and-other-a...
    - humanrebar 12y ago
      Prototumor
    - twp 12y ago
      "There is nothing more permanent than a temporary solution".
    - freshhawk 12y ago
      Hmm, disagree. The prototype doesn't have cancer, it causes the cancer. You're right that it's easier to say but it's saying the wrong thing.
krisdol 12y ago
I am new to the industry and have only really had limited experience. Is it more fair to call these symptoms of dysfunction or business as usual. How do you begin to avoid these problems?
- logicalmind 12y ago
  In my experience, one easy way to determine how good the job is going to be is based on the pecking order of various departments in the company. At a company that does software, the software developers are usually at or near the top. Alternatively, at a marketing company, the marketers are at the top and the software developers may very well be near the bottom. You can gather this information in a number of ways in an interview. Simply ask one of the interviewers where their group exists in the company hierarchy. If you don't want to be explicit, ask to look at the area where the developers sit and work. If the area is well done and people seem happy, you're probably in good shape. If the people are essentially stuffed into a closet or makeshift office/hallway then you might want to run. Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization.
  - groovy2shoes 12y ago
    > Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization. I'd say this is usually good advice, but with a caveat: larger companies with deep chains of command -- even software companies -- typically have "management culture". These types of companies put a lot of pressure on their employees to move into management, even if they're not good at it or they don't want to. These companies reward playing social power games above any other kind of achievement. There will be lots of meetings and not a lot of work being done. There will be no reward for technological innovation or productivity. So, I'd say avoid companies with deep management hierarchies unless 1) you want to go into management or 2) are stoked by the thought of doing the bare minimum.
iMark 12y ago
I was on a project that produced a carcinogenic prototype. The irony was we knew it at the time. Before it was shown to the higher-ups, us developers joked that they'd like the prototype so much they'd come back with a completely unrealistic deadline for the finished product. They did. They wanted it ready in 3 months. I estimated 10. We delivered something - I'd hesitate to call it a finished product - 15 months later, and it was deservedly panned by users. I've spent a lot of time since pondering exactly where it all started to go wrong, and I'm entirely convinced the day we demonstrated the prototype was it.
- kabdib 12y ago
  I saw a startup do a tablet once (a very early one). They'd written a lot of demo code in Smalltalk V. I interviewed there, and turned down an offer. When I was talking with them, I joked that they needed to be careful or they'd wind up shipping their demo code. "Oh, don't worry," they said, "We're going to re-write all of that." A year later they shipped their demo code. A year after that they'd sunk without a trace. (I sometimes wonder if they would have had a chance if I'd joined them. I'm guessing it would have turned out pretty much the same).
  - mikestew 12y ago
    "I sometimes wonder if they would have had a chance if I'd joined them." Maybe if you joined as VP of Engineering or CTO. As an IC, all you're going to do is get dragged down with them, and go home pissed off every night. I've been there. A company has no existing test infrastructure, it's all done by hand, and the build "system" are the binaries that come off a dev's machine? No problem! I'm you're man, I'll have you up and automated in a jiffy. Except that the reason there's no existing infrastructure now is because of general cluelessness, not for lack of expertise. You need someone to hook the hoozit that I lightly touched last time I was there to your new whatzit? Yeah, I'll come in on a contract basis to get it sorted. My first clue should be that none of the other devs who know it better than I came to your rescue. No specs for the thing I'm supposed to integrate with, no documentation as to what it is you actually want, and a PM that sits and watches YouTube videos all day? Yeah, I should have guessed. It's top-down, and if you're not at the top, you're not going to save them from themselves.
ef47d35620c1 12y ago
The 'it depends' answers are good and I would not cite them as evidence of a problem. You employee people who understand what they are talking about and who also understand the importance of an accurate answer. They are probably very good engineers. How does X close a TCP connection... it depends on the operating system in question. What cipher does my browser use when talking to X website... it depends on what ciphers are supported/available and how the client/server are configured. Which router do these packets go through... it depends. Is my password secure... it depends on the string you chose, the hash type used to store the password and who the attacker is and what their resources and time frames are. There is hardly anything absolute in technology/software. And, people who want an absolute answer are only indicating that they do not understand the fundamental complexity issues that we deal with as technologists.
- deleted 12y ago
  [deleted]
- thu 12y ago
  That didn't seem his point. He didn't say at all that the person providing the answer was not competent or being imprecise or whatever negative. The example he gave was different that yours: it was about the software the team is responsible for and about the fact that technical debt has been formed. In dysfunctional organizations, dysfunctional software happens despite of the bright people.
  - a3n 12y ago
    "In dysfunctional organizations, dysfunctional software happens despite of the bright people." A variation of Conway's law. https://en.wikipedia.org/wiki/Conways_Law https://en.wikipedia.org/wiki/Conways_Law
    - zachrose 12y ago
      The potential corollary: "In functional organizations, functional software happens even when the people aren't that bright." Exciting or sad, or both?
AnimalMuppet 12y ago
I once worked with a guy who gave a lot of "it depends" answers. (In fairness, he was the chief scientist, and there was a lot more of valid "it depends" than I wanted to recognize in the way microwaves interact with living tissue.) One day a coworker and I asked him a question, and demanded that he give us a yes or no answer. He thought for a moment, and then replied: "Yes. However..." [Edit: Fixed typo.]
pg_is_a_butt 12y ago
very terribly written article. unreadable. maybe you know so much about dysfunction in software teams because you're the dysfunction re communication.
iamaprogrammer 12y ago
Yeah... that post hit really hard. I got to quit my job or I will go crazy. Dependencies - Check Jane Doe - Check Town Hero - Hey look! It's me! Carcinogenic Prototype - Yeah sure, that's the one Jane Doe is maintaining!
- nahname 12y ago
  The hero is negligent in their duties and intentionally leave an app is a poor state to promote their importance. Are you sure you are the hero?
  - iamaprogrammer 12y ago
    yes I am. Me and all my coworkers, are actually quite good but are doing a shitty job. I'm included. The shitty part is that they pay me above the market.
  - nikatwork 12y ago
    The hero is called in to patch up some flaming wreckage. Mgmt never allocates the resources to actually fix the root cause. Thus the hero is seldom at fault, otherwise they would be Jane Doe. There is a specific type of hero who deliberately creates this situation, but IME they are not the rule. Source: I was That Guy for about 8 years at three different gigs. Now I specifically negotiate no afterhours support in my job interviews, and turn off my phone when on holiday.
crazy1van 12y ago
I disagree a bit with these points. Sure, ideally you'd avoid all of these issues and having these issues several years into a product is certainly bad. But when a new product is just getting off the ground, there is a hefty cost to Doing It Right. I think many successful products would never have launched if a scary number of corners weren't cut to get version 1.0 out the door.
- loup-vaillant 12y ago
  Depends what kind of product. If you're a start-up launching for Christmas, sure. But if it's a contract to deliver some long term custom product at a multi-year deadline (I have worked on such things), then you should definitely not push that prototype. (Actually, the inverse error is often made: trying to get it right the first time around, which is impossible. A prototype is even worse, but least you can test your major ideas, so you can see which are good, and which are hopeless.)
  - ak39 12y ago
    I think the "prototype" analogue borrowed from engineering to software is a bit problematic. I don't believe there is such a thing as a "working prototype" in software. It either works or it doesn't. If it works and that's what business needs in the production environment, then ship it already. Nothing wrong with that. The problems start arising when the issues of scalability and maintenance were not considered part of the prototype development. And this problem does not only affect "carcinogenic prototypes" [sic] ... such problems manifest, and do so regularly, with software projects that had multi-year deadlines and good intentions all-round vis-à-vis scalability and easy maintenance.
- pnathan 12y ago
  These aren't technical issues. These are issues of process and management. I strongly doubt they have much unique to software development systems, to be honest.
loup-vaillant 12y ago
It was a neat prototype, you know. A few loose ends here and there, but a solid proof of concept. Executives loved it. Could see the $_$ in their eyes. So they had us plug the holes and get it to production as fast as possible. Time to market, that sort of things. I recall John didn't like it. Caused quite a stir. But who was he to decide anyway? […] Yeah, that became quite the thing. Lots of new features. Funny thing is, they used to come fast. Not so much lately. Did ask why —were simple features—, but all I got was some bullshit about about "technical debt". Don't give a crap is what they do. No matter, we're entering cash cow mode now. Champaign? […] I'd like to, but John left (something about a "big ball of mud"), and Bob got promoted… That leaves Jane and Sally. Still many glitches to fix, but they will do —not like I have a choice. Besides, they know the billing subsystem inside out, and women are good at cleaning things up, right? Wait, I didn't —are you recording this? […] Phew, that was a close call. Jane warned IT about that OS update, but they kept talking about this "security update". Four days without revenue! Jane and Sally were no good, even Bob couldn't see what was wrong. Had to call John back, you know. Fixed the problem in a pinch. I hear HR made an offer, but I guess there was no changing his mind. Oh well, at least things are back to normal. […] Dunno why, but this systems scares me a little bit.
hcrisp 12y ago
I nominate this author as the Fred Brooks of the 21st century. (Brooks wrote "Mythical Man Month"). These depictions of organizational sickness are spot on. I look forward to hearing more. The Carcinogenic Prototype is what Brooks might have called "the tar pit".
alrs 12y ago
The easiest heuristic is "Are they changing ticketing systems twice per year?" I've never seen a competent team spend any amount of time chasing after ticketing system nirvana.
- icedchai 12y ago
  Sounds about right. I briefly worked at a place migrating from Trac to Jira. Oh, and they still had some stuff in Bugzilla. This team wasn't more than 2 years old.
- bananas 12y ago
  Agree. The last company I worked for used salesforce, email and excel to manage a 1 million line enterprise behemoth. Without much hesitation as I was employed to fix the company, with zero budget we fired up trac and integrated it with AD and their reporting stuff. Now this was frowned upon (open source!?!?! Never!) but it delivered real results and traceability. Now a year down the line and 5000 tickets later they finally accepted the value and assigned a PM to look at the company processes which were clearly broken. A month later a totally broken copy of JIRA appeared whilst I was on holiday with the old process that was broken crudely crammed into it with no reporting, no backup/restore and no amount of crazy spared. Yep clearly the problem was the issue tracker not the process. Now $32000 a year in protection money from Atlassian hits the upper management team (as well as the process brick wall which bottlenecks everything on the PM) so they decided that the problem is: "We need another ticketing system. Sales force is looking good if we can just customise it for our process." So I quit with much "fuuuuuuuu..."
penguindev 12y ago
> As with most things in software engineering the technical problems are symptoms of the economic causes. FTFY. Seriously - #2 & #3, not enough money for a 'co pilot'. And for 2, at least there is some ostensible owner of the module, rather than a maintenance by committee which can be even shittier. #1, maybe not cost effective to move, or would be opportunity cost. #4, I'm sympathetic to not shipping the prototype, but ever hear of first mover / time to market? obsolescence is one of the biggest risks in development[1]. Plan on shipping it and rewriting it in bite size chunks ... forever! Anyway, it was a nice read. 'mortgage driven development' hah. [1] In the Mythical Man Month, these are both mentioned, even though they contradict each other, and many more developers seem to only quote the 'don't ship the prototype'.

103 points by buzzkills 12y ago

I've been stuck maintaining a carcinogenic prototype for the last 2 months. While I don't like dealing with the maintenance nightmare, I do really like that term. I'd never heard it before, but I knew what it was as soon as I read it -- the text in that section only confirmed my suspicion.

knieveltech 12y ago
Yeah, this definitely feels like a permanent addition to the lexicon. It crisply describes at least two systems I'm maintaining currently.
- wmeredith 12y ago
  If it gets canonized, I'd much prefer "cancerous prototype" because it's easier to say.
  - MaysonL 12y ago
    "metastatic prototype"
    - cpr 12y ago
      You mean metatastic prototype?
  - schnevets 12y ago
    I like this better too - the prototype will inevitably grow from its humble origins, and with every new feature that is hastily thrown in, the application becomes ever so much riskier.
  - jmcphers 12y ago
    Also known as: Quick Hack, Kleenex Code, Disposable Code, Scripting, Killer Demo, Permanent Prototype, Boomtown. [1] http://blog.codinghorror.com/the-big-ball-of-mud-and-other-architectural-disasters/ http://blog.codinghorror.com/the-big-ball-of-mud-and-other-a...
  - humanrebar 12y ago
    Prototumor
  - twp 12y ago
    "There is nothing more permanent than a temporary solution".
  - freshhawk 12y ago
    Hmm, disagree. The prototype doesn't have cancer, it causes the cancer. You're right that it's easier to say but it's saying the wrong thing.

krisdol 12y ago

I am new to the industry and have only really had limited experience. Is it more fair to call these symptoms of dysfunction or business as usual. How do you begin to avoid these problems?

logicalmind 12y ago
In my experience, one easy way to determine how good the job is going to be is based on the pecking order of various departments in the company. At a company that does software, the software developers are usually at or near the top. Alternatively, at a marketing company, the marketers are at the top and the software developers may very well be near the bottom. You can gather this information in a number of ways in an interview. Simply ask one of the interviewers where their group exists in the company hierarchy. If you don't want to be explicit, ask to look at the area where the developers sit and work. If the area is well done and people seem happy, you're probably in good shape. If the people are essentially stuffed into a closet or makeshift office/hallway then you might want to run. Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization.
- groovy2shoes 12y ago
  > Generally, try to work for a company that makes money on software or technology. A company that makes money on other products or services is likely to treat software developers as a low-level function of the organization. I'd say this is usually good advice, but with a caveat: larger companies with deep chains of command -- even software companies -- typically have "management culture". These types of companies put a lot of pressure on their employees to move into management, even if they're not good at it or they don't want to. These companies reward playing social power games above any other kind of achievement. There will be lots of meetings and not a lot of work being done. There will be no reward for technological innovation or productivity. So, I'd say avoid companies with deep management hierarchies unless 1) you want to go into management or 2) are stoked by the thought of doing the bare minimum.

iMark 12y ago

I was on a project that produced a carcinogenic prototype. The irony was we knew it at the time. Before it was shown to the higher-ups, us developers joked that they'd like the prototype so much they'd come back with a completely unrealistic deadline for the finished product. They did. They wanted it ready in 3 months. I estimated 10. We delivered something - I'd hesitate to call it a finished product - 15 months later, and it was deservedly panned by users. I've spent a lot of time since pondering exactly where it all started to go wrong, and I'm entirely convinced the day we demonstrated the prototype was it.

kabdib 12y ago
I saw a startup do a tablet once (a very early one). They'd written a lot of demo code in Smalltalk V. I interviewed there, and turned down an offer. When I was talking with them, I joked that they needed to be careful or they'd wind up shipping their demo code. "Oh, don't worry," they said, "We're going to re-write all of that." A year later they shipped their demo code. A year after that they'd sunk without a trace. (I sometimes wonder if they would have had a chance if I'd joined them. I'm guessing it would have turned out pretty much the same).
- mikestew 12y ago
  "I sometimes wonder if they would have had a chance if I'd joined them." Maybe if you joined as VP of Engineering or CTO. As an IC, all you're going to do is get dragged down with them, and go home pissed off every night. I've been there. A company has no existing test infrastructure, it's all done by hand, and the build "system" are the binaries that come off a dev's machine? No problem! I'm you're man, I'll have you up and automated in a jiffy. Except that the reason there's no existing infrastructure now is because of general cluelessness, not for lack of expertise. You need someone to hook the hoozit that I lightly touched last time I was there to your new whatzit? Yeah, I'll come in on a contract basis to get it sorted. My first clue should be that none of the other devs who know it better than I came to your rescue. No specs for the thing I'm supposed to integrate with, no documentation as to what it is you actually want, and a PM that sits and watches YouTube videos all day? Yeah, I should have guessed. It's top-down, and if you're not at the top, you're not going to save them from themselves.

ef47d35620c1 12y ago

The 'it depends' answers are good and I would not cite them as evidence of a problem. You employee people who understand what they are talking about and who also understand the importance of an accurate answer. They are probably very good engineers. How does X close a TCP connection... it depends on the operating system in question. What cipher does my browser use when talking to X website... it depends on what ciphers are supported/available and how the client/server are configured. Which router do these packets go through... it depends. Is my password secure... it depends on the string you chose, the hash type used to store the password and who the attacker is and what their resources and time frames are. There is hardly anything absolute in technology/software. And, people who want an absolute answer are only indicating that they do not understand the fundamental complexity issues that we deal with as technologists.

deleted 12y ago
[deleted]
thu 12y ago
That didn't seem his point. He didn't say at all that the person providing the answer was not competent or being imprecise or whatever negative. The example he gave was different that yours: it was about the software the team is responsible for and about the fact that technical debt has been formed. In dysfunctional organizations, dysfunctional software happens despite of the bright people.
- a3n 12y ago
  "In dysfunctional organizations, dysfunctional software happens despite of the bright people." A variation of Conway's law. https://en.wikipedia.org/wiki/Conways_Law https://en.wikipedia.org/wiki/Conways_Law
  - zachrose 12y ago
    The potential corollary: "In functional organizations, functional software happens even when the people aren't that bright." Exciting or sad, or both?

AnimalMuppet 12y ago

I once worked with a guy who gave a lot of "it depends" answers. (In fairness, he was the chief scientist, and there was a lot more of valid "it depends" than I wanted to recognize in the way microwaves interact with living tissue.) One day a coworker and I asked him a question, and demanded that he give us a yes or no answer. He thought for a moment, and then replied: "Yes. However..." [Edit: Fixed typo.]

pg_is_a_butt 12y ago

very terribly written article. unreadable. maybe you know so much about dysfunction in software teams because you're the dysfunction re communication.

iamaprogrammer 12y ago

Yeah... that post hit really hard. I got to quit my job or I will go crazy. Dependencies - Check Jane Doe - Check Town Hero - Hey look! It's me! Carcinogenic Prototype - Yeah sure, that's the one Jane Doe is maintaining!

nahname 12y ago
The hero is negligent in their duties and intentionally leave an app is a poor state to promote their importance. Are you sure you are the hero?
- iamaprogrammer 12y ago
  yes I am. Me and all my coworkers, are actually quite good but are doing a shitty job. I'm included. The shitty part is that they pay me above the market.
- nikatwork 12y ago
  The hero is called in to patch up some flaming wreckage. Mgmt never allocates the resources to actually fix the root cause. Thus the hero is seldom at fault, otherwise they would be Jane Doe. There is a specific type of hero who deliberately creates this situation, but IME they are not the rule. Source: I was That Guy for about 8 years at three different gigs. Now I specifically negotiate no afterhours support in my job interviews, and turn off my phone when on holiday.

crazy1van 12y ago

I disagree a bit with these points. Sure, ideally you'd avoid all of these issues and having these issues several years into a product is certainly bad. But when a new product is just getting off the ground, there is a hefty cost to Doing It Right. I think many successful products would never have launched if a scary number of corners weren't cut to get version 1.0 out the door.

loup-vaillant 12y ago
Depends what kind of product. If you're a start-up launching for Christmas, sure. But if it's a contract to deliver some long term custom product at a multi-year deadline (I have worked on such things), then you should definitely not push that prototype. (Actually, the inverse error is often made: trying to get it right the first time around, which is impossible. A prototype is even worse, but least you can test your major ideas, so you can see which are good, and which are hopeless.)
- ak39 12y ago
  I think the "prototype" analogue borrowed from engineering to software is a bit problematic. I don't believe there is such a thing as a "working prototype" in software. It either works or it doesn't. If it works and that's what business needs in the production environment, then ship it already. Nothing wrong with that. The problems start arising when the issues of scalability and maintenance were not considered part of the prototype development. And this problem does not only affect "carcinogenic prototypes" [sic] ... such problems manifest, and do so regularly, with software projects that had multi-year deadlines and good intentions all-round vis-à-vis scalability and easy maintenance.
pnathan 12y ago
These aren't technical issues. These are issues of process and management. I strongly doubt they have much unique to software development systems, to be honest.

loup-vaillant 12y ago

It was a neat prototype, you know. A few loose ends here and there, but a solid proof of concept. Executives loved it. Could see the $_$ in their eyes. So they had us plug the holes and get it to production as fast as possible. Time to market, that sort of things. I recall John didn't like it. Caused quite a stir. But who was he to decide anyway? […] Yeah, that became quite the thing. Lots of new features. Funny thing is, they used to come fast. Not so much lately. Did ask why —were simple features—, but all I got was some bullshit about about "technical debt". Don't give a crap is what they do. No matter, we're entering cash cow mode now. Champaign? […] I'd like to, but John left (something about a "big ball of mud"), and Bob got promoted… That leaves Jane and Sally. Still many glitches to fix, but they will do —not like I have a choice. Besides, they know the billing subsystem inside out, and women are good at cleaning things up, right? Wait, I didn't —are you recording this? […] Phew, that was a close call. Jane warned IT about that OS update, but they kept talking about this "security update". Four days without revenue! Jane and Sally were no good, even Bob couldn't see what was wrong. Had to call John back, you know. Fixed the problem in a pinch. I hear HR made an offer, but I guess there was no changing his mind. Oh well, at least things are back to normal. […] Dunno why, but this systems scares me a little bit.

hcrisp 12y ago

I nominate this author as the Fred Brooks of the 21st century. (Brooks wrote "Mythical Man Month"). These depictions of organizational sickness are spot on. I look forward to hearing more. The Carcinogenic Prototype is what Brooks might have called "the tar pit".

alrs 12y ago

The easiest heuristic is "Are they changing ticketing systems twice per year?" I've never seen a competent team spend any amount of time chasing after ticketing system nirvana.

icedchai 12y ago
Sounds about right. I briefly worked at a place migrating from Trac to Jira. Oh, and they still had some stuff in Bugzilla. This team wasn't more than 2 years old.
bananas 12y ago
Agree. The last company I worked for used salesforce, email and excel to manage a 1 million line enterprise behemoth. Without much hesitation as I was employed to fix the company, with zero budget we fired up trac and integrated it with AD and their reporting stuff. Now this was frowned upon (open source!?!?! Never!) but it delivered real results and traceability. Now a year down the line and 5000 tickets later they finally accepted the value and assigned a PM to look at the company processes which were clearly broken. A month later a totally broken copy of JIRA appeared whilst I was on holiday with the old process that was broken crudely crammed into it with no reporting, no backup/restore and no amount of crazy spared. Yep clearly the problem was the issue tracker not the process. Now $32000 a year in protection money from Atlassian hits the upper management team (as well as the process brick wall which bottlenecks everything on the PM) so they decided that the problem is: "We need another ticketing system. Sales force is looking good if we can just customise it for our process." So I quit with much "fuuuuuuuu..."

penguindev 12y ago

> As with most things in software engineering the technical problems are symptoms of the economic causes. FTFY. Seriously - #2 & #3, not enough money for a 'co pilot'. And for 2, at least there is some ostensible owner of the module, rather than a maintenance by committee which can be even shittier. #1, maybe not cost effective to move, or would be opportunity cost. #4, I'm sympathetic to not shipping the prototype, but ever hear of first mover / time to market? obsolescence is one of the biggest risks in development[1]. Plan on shipping it and rewriting it in bite size chunks ... forever! Anyway, it was a nice read. 'mortgage driven development' hah. [1] In the Mythical Man Month, these are both mentioned, even though they contradict each other, and many more developers seem to only quote the 'don't ship the prototype'.