8 ms·

They neglect maintenance because we reward resolving outages and fighting fires more than we do good stewardship. Imagine the following scenario. Acme Co. has

They neglect maintenance because we reward resolving outages and fighting fires more than we do good stewardship. Imagine the following scenario. Acme Co. has some piece of software that is critical to the operation for their business. Dave is in charge of the software system. The software crashes often but Dave is always there to save the day. Dave often has to stay up all night and is seen fixing problems at the final hour. Management takes notice of this and often sends out emails congratulating Dave for his hard work. One day Dave leaves, and is replaced by Amy. Amy takes over the system and slowly and methodically fixes all the bugs that were causing the outages in the software.¹ Over time the code becomes more reliable, till eventually it runs smoothly. Now what would management say if you asked which programmer was better. They'd likely say: "Amy is pretty good, but boy that guy Dave was a real rockstar!" When you reward a behaviour, you will get more of it. If you want software maintenance, then you need to reward it. But often, perversely we reward its opposite. I'm working mostly as an ops person these days, and if I do my job well it makes my contribution look invisible. When things are going right, there won't be anything to notice. 1: In other words, code maintenance.

BubRoss 7y ago
Fire fighters are heros but the man who designs a house that never burns down is forgotten.
- eloisant 7y ago
  ...and in our industry, the fire fighters praised by management are often the one who designed the house like a tinderbox.
Jtsummers 7y ago
Relevant paper on this: https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_....
michannne 7y ago
Even in the case of a suitable rewards system, one still needs the skill to be able to find and fix bugs that cause systemic issues. Many times, a company opts to get the developers who are good enough - they can build the system, and then they hand it off to some Ops team that maintains, and good luck trying to get them to fix any of the bugs introduced over the course of the project. What should be rewarded is the skill necessary to architect software in such a way that it is easy to maintain. I often say to myself, does this decrease the effort needed by the reporting team to create reports? Does it isolate features and make it easy for devs to find the source of issues and fix them? Is it modular enough that users can expect a quick turnaround time for new features? What I've found is that many developers, even those who believe they should be seniors and above, are unable to think from this perspective, or frequently think it's "impossible" to make software that addresses and solves the issues of all groups. In my experience, it is never impossible, it is all about having the right architecture, which can takes weeks to fully think up and realize, and may still change afterwards, but the design of the software is what I usually choose to reward.
- gerbilly 7y ago
  > frequently think it's "impossible" to make software that addresses and solves the issues of all groups. I find this increasingly to be the case. The fashion now is to do everything with MVPs and sprints. The big hidden assumption with this style of development is that any problem can be solved using incremental 'local searches'¹ through the design space. This is definitely not true, especially for co-optimization problems like you describe above². 1: https://en.wikipedia.org/wiki/Hill_climbing https://en.wikipedia.org/wiki/Hill_climbing 2: "Solving the issues for all groups" appears like a meta co-optimization problem.
bloat 7y ago
In all the places I've worked over the last decade Dave would have many, many incident reports against his name, and all the root cause investigations would not look good for him either.
- jolmg 7y ago
  I wonder if the difference lies in whether the business specializes in software. In a business that doesn't, I don't think they would invest much in investigating. Dave might be the only person qualified to do such an investigation, after all. It could also be a matter of the ratio between tech work to be done and tech workers available. In an IT department that has too much pending work, it might be more worthwhile to implement quick crap solutions than to go back and properly fix stuff. You could call this a failure in properly funding such a department, but this is viable to happen in a business where the executives (or whoever controls the funding) don't really understand the difference between crap tech work and good tech work.
- rconti 7y ago
  This would presume that the person doing the firefighting is the original author of the software. That you're always on call specifically for your own code. In most ops roles, Dave would be fixing someone else's broken-ass system, and anyone else in the rotation would have a similar number of "incidents" against them.
ASalazarMX 7y ago
Amy would have seemed as rockstar as Dave in the beginning, as she had to do all his work plus diagnostics plus analysis and fixing. If she made sure to inform what is she fixing, management would appreciate her more. There are nefarious managers, but most times it's our own failings to communicate that makes us invisible.
- Razengan 7y ago
  > it's our own failings to communicate that makes us invisible. It's not a failing to be introverted and not desire attention/spotlight. > If she made sure to inform what is she fixing, management would appreciate her more. Most people usually get annoyed when you tell them what you're doing for them.
  - recursive 7y ago
    > It's not a failing to be introverted and not desire attention/spotlight. It is a failing if you care about getting recognition. Self-promotion isn't the only reason for recognition. For instance you might want to promote proactive maintenance instead of reactive repairs. > Most people usually get annoyed when you tell them what you're doing for them. That is very different from my personal experience.
    - tstrimple 7y ago
      Anecdotally I have to agree with you. When I worked at Microsoft as a Cloud Solution Architect, my job was to help companies adopt the Microsoft cloud. There were some companies at a fairly famous startup accelerator which everyone here is familiar with who I was already engaged with when a new colleague also started helping companies out of this accelerator. I was quietly doing my job and helping solve problems, he made a big fuss about how he got Microsoft's foot in the door at this accelerator. Naturally he got tons of accolades and even a bonus for his work. This need to constantly be selling yourself internally was ultimately what lead to my departure from Microsoft. I'm here to solve problems, not to talk myself up as the savior of the world. It's also why I much prefer working with startups. Your work is much more evident and you aren't constantly having to tell everyone what you've done.
teunispeters 7y ago
Over several decades I have been burnt many, many times trying to be the "Amy". because Dave isn't blamed for the incident reports - either the original reporter is or no blame is assigned. Dave gets all the accolades though. Human nature at its "finest". I now work at a place where that's no longer an issue for me because "blaming the messenger" isn't a management culture where I am now.
coldcode 7y ago
I worked at a place where the chief architect was exactly Dave. Our core app was a pile of crap he had written that routinely got gummed up and he heroically "fixed it" and all the execs raved about him. The poor sucks who supported the app (and me as well) knew it was because the app leaked like a sieve but was fixable, but he wouldn't allow it because then he would have to admit it was his fault.
malms 7y ago
plus amy would be fired because she has no work anymore
- reroute1 7y ago
  Or conversely she could be rewarded for fixing the underlying problems and given a new, possibly similar task.
brightball 7y ago
I worked at a place where I was able to focus on maintainability for a few months. We were seeing about 30 PagerDuty alerts per month. Through a combination of bug resolution and auto scaling, we got it to an average of less than two pages a month. Moving to AWS Aurora removed the final 2. Eventually the entire system self supported...and was retired a year later.
crtlaltdel 7y ago
i started my career in electrical control systems (relays, mcc design, etc) and have a very different take on maintaining a system as a result. we maintained systems as a matter of safety and productivity. this isn’t just a truck roll to check terminations and maybe do a thermal scan, but revisiting the system design when a change was ordered. will this create a safety concern? will it cause a production stop? will the lines go down? after a number of years in (mostly) software i’ve observed less tolerance for that level of analysis. i understand why it can/will impact velocity, and that the cost to change software (and time) is a fraction of that with physical systems...but still...a part of me winces whenever im like “screw it, im merging this sans impact analysis” ;) has anyone else had a similar experience?
an_d_rew 7y ago
Amen. My experience, exactly.

BubRoss 7y ago

Fire fighters are heros but the man who designs a house that never burns down is forgotten.

eloisant 7y ago
...and in our industry, the fire fighters praised by management are often the one who designed the house like a tinderbox.

Jtsummers 7y ago

Relevant paper on this: https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf https://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_....

michannne 7y ago

Even in the case of a suitable rewards system, one still needs the skill to be able to find and fix bugs that cause systemic issues. Many times, a company opts to get the developers who are good enough - they can build the system, and then they hand it off to some Ops team that maintains, and good luck trying to get them to fix any of the bugs introduced over the course of the project. What should be rewarded is the skill necessary to architect software in such a way that it is easy to maintain. I often say to myself, does this decrease the effort needed by the reporting team to create reports? Does it isolate features and make it easy for devs to find the source of issues and fix them? Is it modular enough that users can expect a quick turnaround time for new features? What I've found is that many developers, even those who believe they should be seniors and above, are unable to think from this perspective, or frequently think it's "impossible" to make software that addresses and solves the issues of all groups. In my experience, it is never impossible, it is all about having the right architecture, which can takes weeks to fully think up and realize, and may still change afterwards, but the design of the software is what I usually choose to reward.

gerbilly 7y ago
> frequently think it's "impossible" to make software that addresses and solves the issues of all groups. I find this increasingly to be the case. The fashion now is to do everything with MVPs and sprints. The big hidden assumption with this style of development is that any problem can be solved using incremental 'local searches'¹ through the design space. This is definitely not true, especially for co-optimization problems like you describe above². 1: https://en.wikipedia.org/wiki/Hill_climbing https://en.wikipedia.org/wiki/Hill_climbing 2: "Solving the issues for all groups" appears like a meta co-optimization problem.

bloat 7y ago

In all the places I've worked over the last decade Dave would have many, many incident reports against his name, and all the root cause investigations would not look good for him either.

jolmg 7y ago
I wonder if the difference lies in whether the business specializes in software. In a business that doesn't, I don't think they would invest much in investigating. Dave might be the only person qualified to do such an investigation, after all. It could also be a matter of the ratio between tech work to be done and tech workers available. In an IT department that has too much pending work, it might be more worthwhile to implement quick crap solutions than to go back and properly fix stuff. You could call this a failure in properly funding such a department, but this is viable to happen in a business where the executives (or whoever controls the funding) don't really understand the difference between crap tech work and good tech work.
rconti 7y ago
This would presume that the person doing the firefighting is the original author of the software. That you're always on call specifically for your own code. In most ops roles, Dave would be fixing someone else's broken-ass system, and anyone else in the rotation would have a similar number of "incidents" against them.

ASalazarMX 7y ago

Amy would have seemed as rockstar as Dave in the beginning, as she had to do all his work plus diagnostics plus analysis and fixing. If she made sure to inform what is she fixing, management would appreciate her more. There are nefarious managers, but most times it's our own failings to communicate that makes us invisible.

Razengan 7y ago
> it's our own failings to communicate that makes us invisible. It's not a failing to be introverted and not desire attention/spotlight. > If she made sure to inform what is she fixing, management would appreciate her more. Most people usually get annoyed when you tell them what you're doing for them.
- recursive 7y ago
  > It's not a failing to be introverted and not desire attention/spotlight. It is a failing if you care about getting recognition. Self-promotion isn't the only reason for recognition. For instance you might want to promote proactive maintenance instead of reactive repairs. > Most people usually get annoyed when you tell them what you're doing for them. That is very different from my personal experience.
  - tstrimple 7y ago
    Anecdotally I have to agree with you. When I worked at Microsoft as a Cloud Solution Architect, my job was to help companies adopt the Microsoft cloud. There were some companies at a fairly famous startup accelerator which everyone here is familiar with who I was already engaged with when a new colleague also started helping companies out of this accelerator. I was quietly doing my job and helping solve problems, he made a big fuss about how he got Microsoft's foot in the door at this accelerator. Naturally he got tons of accolades and even a bonus for his work. This need to constantly be selling yourself internally was ultimately what lead to my departure from Microsoft. I'm here to solve problems, not to talk myself up as the savior of the world. It's also why I much prefer working with startups. Your work is much more evident and you aren't constantly having to tell everyone what you've done.

teunispeters 7y ago

Over several decades I have been burnt many, many times trying to be the "Amy". because Dave isn't blamed for the incident reports - either the original reporter is or no blame is assigned. Dave gets all the accolades though. Human nature at its "finest". I now work at a place where that's no longer an issue for me because "blaming the messenger" isn't a management culture where I am now.

coldcode 7y ago

I worked at a place where the chief architect was exactly Dave. Our core app was a pile of crap he had written that routinely got gummed up and he heroically "fixed it" and all the execs raved about him. The poor sucks who supported the app (and me as well) knew it was because the app leaked like a sieve but was fixable, but he wouldn't allow it because then he would have to admit it was his fault.

malms 7y ago

plus amy would be fired because she has no work anymore

reroute1 7y ago
Or conversely she could be rewarded for fixing the underlying problems and given a new, possibly similar task.

brightball 7y ago

I worked at a place where I was able to focus on maintainability for a few months. We were seeing about 30 PagerDuty alerts per month. Through a combination of bug resolution and auto scaling, we got it to an average of less than two pages a month. Moving to AWS Aurora removed the final 2. Eventually the entire system self supported...and was retired a year later.

crtlaltdel 7y ago

i started my career in electrical control systems (relays, mcc design, etc) and have a very different take on maintaining a system as a result. we maintained systems as a matter of safety and productivity. this isn’t just a truck roll to check terminations and maybe do a thermal scan, but revisiting the system design when a change was ordered. will this create a safety concern? will it cause a production stop? will the lines go down? after a number of years in (mostly) software i’ve observed less tolerance for that level of analysis. i understand why it can/will impact velocity, and that the cost to change software (and time) is a fraction of that with physical systems...but still...a part of me winces whenever im like “screw it, im merging this sans impact analysis” ;) has anyone else had a similar experience?

an_d_rew 7y ago

Amen. My experience, exactly.