4 ms·

Early in my career, I got stuck doing a lot of software maintenance on legacy systems (I suspect this permanently stunted my professional growth, but this was d

by jcadam 7y ago

Early in my career, I got stuck doing a lot of software maintenance on legacy systems (I suspect this permanently stunted my professional growth, but this was during the great recession and I took whatever work I could find). It's a hard rut to get out of - I spent a lot of time developing relevant skills in my spare time. Anyway, one place I worked had a 10 page procedure for building the system. It involved running several scripts, logged into 2 or 3 different servers (we were running Solaris 8 - which was already quite old by then) as different users. About half the procedure was written down, the rest was unspoken and passed down via oral tradition. A particularly intelligent and motivated guy once took it upon himself to unravel the whole mess and come up with a sane build solution. Unfortunately, his quest was not approved by management, the time taken away from bug fixing caused him to fall behind on "points" (i.e., number of tickets closed in the bug tracker), and he was laid off. So we stuck to the old system that sort of worked, most of the time. Now, one of the user accounts that was required to build a certain module belonged to a person that no longer worked at the company (let's call him Steve - I don't remember his real name). We were certain there was a permissions or environment issue somewhere, but were unable to track down the specific reason why only Steve's user account could successfully build said module. We were successfully ignoring this obviously absurd situation until we got a new sysadmin.... The new sysadmin was unfortunately a very motivated guy who wanted to set things in order. He reviewed all of the user accounts, noticed "Steve" no longer worked for the company, and deleted his account. We were unable to build until the backup containing Steve's account was restored. We never fixed the FUBARed build procedure, mostly because our customer (this was a defense contract, so the govt was the customer) didn't want to pay for such an effort - every thing we did had to be tied back to a specific req number in our system requirements specification (SRS). Fixing technical debt was not covered in the SRS, so it never got fixed. Eventually I was caught up in a mass layoff and went on to better things :)

ScottFree 7y ago
What I got from your story was: 1) Long term software maintenance should not be treated like help desk ticket work nor should it be dumped off on junior programmers. Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job. 2) Make sure fixing technical debt is covered in the SRS.
- jcadam 7y ago
  > Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job. That's a career death sentence. I suppose if I was retired and wanted a part-time gig for extra money/amusement, I'd do it. But one thing I learned from working maintenance early in my career was... don't do maintenance work.
  - GFischer 7y ago
    It is. Also, people don't want to do it for any amount of money. I quit a previous job and took less money because I got moved to maintain the crumbling Forte4GL core system - I did get a handsome raise for that, but the writing was also on the wall as they were also working to replace it. Migration project has been ongoing for 8 years and is nowhere near done, I might have been able to retire before it's completed though. It was also soul-sucking.
    - sokoloff 7y ago
      People don’t want to do it for any amount of money that is offered for that work. There is an amount of money that will clear that market, though.
      - scarface74 7y ago
        Only if you optimize for your current salary instead of optimizing for your future career prospects. There isn't a reasonable amount of money (it would have to be twice my current salary) for me to accept a job doing VB6 or any other job that would endanger my future employment prospects.
        bluGill 7y ago
        If I can retire one my savings after the job is done I don't care about future prospects. I'd rather work on [insert any hobby with no commercial value] at home than go to work.
        7y ago
ownagefool 7y ago
There's probably a bit more nuance to this. I've worked for Gov, Consultancies and Private sector clients. They all have shared part of the blame at some point. But, I've specifically seen the companies charge £75k for firewall changes or sever upgrades, when the reality was they're already paid £10,000,000 a year manage servers that a couple of good puppet admins could have done. Point being, it's sometimes not about not wanting to pay, but simply about not wanting to pay for something that should have already been done. Of course, this all comes back to the fact they gleefully oursorced their tech competency, based on the recommendation of consultancies, based on their lack of long term strategic management, so they apparently get what they deserve. It's a shame that Gov is spending tax money though.
- sailfast 7y ago
  Outsourcing of technical capability (especially prevalent in government) bites you hard when you lose the ability to work directly with your contract teams and have to rely on their direction. (Especially if the management is non-technical) There’s a clear conflict of interest on the part of the contractor, and you need to be able to set their direction competently. I would recommend always having an in-house capability, even if it’s just an in-house consultancy, so that you don’t run into these kinds of issues and can provide adequate direction / oversight.
  - jcadam 7y ago
    The GS payscale is not set up to be able to hire and retain highly skilled technical experts. Even if I were to be hired at the GS-15 level, I'd take a pay cut from what I earn as a contractor. I once worked on a contract in a Systems Engineering and Technical Assistance (SETA) capacity, which was both an odd and eye-opening experience - basically advising the govt customer, as a technical expert, on contractors' proposals and sitting in on reviews and such.
    - sailfast 7y ago
      That's a good fallback option, though it's still difficult to control turnover and feel like you are aligned in terms of value using that approach. You'd want to find somebody that you trust and bring them on in that special capacity, rather than just putting an RFP on the street for a SETA advisor. Agreed on the GS payscale and its ability to keep up with high-end private sector pay though I also think that the entire hiring process may be more of a barrier than the pay itself. All that said, a non-supervisory GS-15 in the DC area (18% locality pay bump - more if you're SF or NY) is maxed out in the 170s which isn't shabby!! When you combine that wage with generous retirement matching options, job security, good work/life balance, and excellent health insurance at well negotiated rates it ends up being a pretty attractive offer. Furthermore, there are other government agencies and roles that have their own specialized payscale outside of the GS scale. This can include financial regulators (higher pay to avoid revolving door problems) as well as "cybersecurity" billets across different places.
rantwasp 7y ago
this story is so familiar that it’s not even funny. I also have seen this while the project is developed. No incentives for fixing stuff that would exponentially increase the productivity of everyone - just keep adding shit until you move on. Pretend like learning the obscurities of a home-brewed/hastily hacked mess matter.
- naringas 7y ago
  a few years of that until one day, for example Apple realeases Catalina?
gwbas1c 7y ago
> I suspect this permanently stunted my professional growth, but this was during the great recession and I took whatever work I could find From reading your post, I think this was excellent experience in understanding office politics; and in software development. One of the ways to learn how to write maintainable software is to maintain someone else's software!
- kaycebasques 7y ago
  I’m also curious why OP should think that this might permanently stunt a career.
  - non-entity 7y ago
    OP mentioned the work was on legacy systems, and many of the top jobs that everyone want are likely going to be doing new development with new(ish) tech. I'd imagine especially doing legacy work at a non-tech company will not look very good on your resume to a tech company (whether that be a startup, small software firm, or big n) Just my guess
    - jcadam 7y ago
      > ... doing legacy work at a non-tech company will not look very good on your resume to a tech company... This, more or less. Maintenance also tends to be a very low-visibility activity at most companies as well. Promotions go to folks who can make a name for themselves working on the company's hot new project. It took a while to find a job using modern tech when I had a resume full of things like Solaris 8, Ada95, ClearCase, and Oracle. I finally found a company to give me a chance doing new development using modern tech (well, Java, at least) back in 2012 (in my early thirties). My career has been doing much better since, but I feel like I'm far behind where I should be for someone my age (not in skill, mind you, but in pay, role, status, etc).
FoomFries 7y ago
I find the trick is to do maintenance while doing meaningful (as described by the powers that be) work. So when a new feature or bug fix is asked for, add another 15% to the time required, and clean up the code base a bit. The knee-jerk, which rarely works, is to demand time to refactor everything or ignore everything until it breaks irreparably. I think in your case, simplifying a script by a dozen lines a week or so could have been done without taking away "points". Abandoning the status quo and committing too much time will inevitably be looked down on in most industries, however.
- bluGill 7y ago
  The poster was doing a military contract, not working for his own company. His companies money - and thus his bonus - is in putting in hours to do what is requested. Saving money means they get paid less hours now the project is smaller and so the bonus is less because they are paid less. In private industry it can (but often does not) mean that you can get more profit and thus a larger bonus by saving your company money long term. However in the world of contractors saving the client money this is not rewarded.
natalyarostova 7y ago
> permanently stunted my professional growth This is such a counter-productive attitude.

by jcadam 7y ago

ScottFree 7y ago

What I got from your story was: 1) Long term software maintenance should not be treated like help desk ticket work nor should it be dumped off on junior programmers. Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job. 2) Make sure fixing technical debt is covered in the SRS.

jcadam 7y ago
> Every company should have at least one guy who enjoys maintaining legacy systems and that should be his sole job. That's a career death sentence. I suppose if I was retired and wanted a part-time gig for extra money/amusement, I'd do it. But one thing I learned from working maintenance early in my career was... don't do maintenance work.
- GFischer 7y ago
  It is. Also, people don't want to do it for any amount of money. I quit a previous job and took less money because I got moved to maintain the crumbling Forte4GL core system - I did get a handsome raise for that, but the writing was also on the wall as they were also working to replace it. Migration project has been ongoing for 8 years and is nowhere near done, I might have been able to retire before it's completed though. It was also soul-sucking.
  - sokoloff 7y ago
    People don’t want to do it for any amount of money that is offered for that work. There is an amount of money that will clear that market, though.
    - scarface74 7y ago
      Only if you optimize for your current salary instead of optimizing for your future career prospects. There isn't a reasonable amount of money (it would have to be twice my current salary) for me to accept a job doing VB6 or any other job that would endanger my future employment prospects.
      - bluGill 7y ago
        If I can retire one my savings after the job is done I don't care about future prospects. I'd rather work on [insert any hobby with no commercial value] at home than go to work.
      - 7y ago

ownagefool 7y ago

There's probably a bit more nuance to this. I've worked for Gov, Consultancies and Private sector clients. They all have shared part of the blame at some point. But, I've specifically seen the companies charge £75k for firewall changes or sever upgrades, when the reality was they're already paid £10,000,000 a year manage servers that a couple of good puppet admins could have done. Point being, it's sometimes not about not wanting to pay, but simply about not wanting to pay for something that should have already been done. Of course, this all comes back to the fact they gleefully oursorced their tech competency, based on the recommendation of consultancies, based on their lack of long term strategic management, so they apparently get what they deserve. It's a shame that Gov is spending tax money though.

sailfast 7y ago
Outsourcing of technical capability (especially prevalent in government) bites you hard when you lose the ability to work directly with your contract teams and have to rely on their direction. (Especially if the management is non-technical) There’s a clear conflict of interest on the part of the contractor, and you need to be able to set their direction competently. I would recommend always having an in-house capability, even if it’s just an in-house consultancy, so that you don’t run into these kinds of issues and can provide adequate direction / oversight.
- jcadam 7y ago
  The GS payscale is not set up to be able to hire and retain highly skilled technical experts. Even if I were to be hired at the GS-15 level, I'd take a pay cut from what I earn as a contractor. I once worked on a contract in a Systems Engineering and Technical Assistance (SETA) capacity, which was both an odd and eye-opening experience - basically advising the govt customer, as a technical expert, on contractors' proposals and sitting in on reviews and such.
  - sailfast 7y ago
    That's a good fallback option, though it's still difficult to control turnover and feel like you are aligned in terms of value using that approach. You'd want to find somebody that you trust and bring them on in that special capacity, rather than just putting an RFP on the street for a SETA advisor. Agreed on the GS payscale and its ability to keep up with high-end private sector pay though I also think that the entire hiring process may be more of a barrier than the pay itself. All that said, a non-supervisory GS-15 in the DC area (18% locality pay bump - more if you're SF or NY) is maxed out in the 170s which isn't shabby!! When you combine that wage with generous retirement matching options, job security, good work/life balance, and excellent health insurance at well negotiated rates it ends up being a pretty attractive offer. Furthermore, there are other government agencies and roles that have their own specialized payscale outside of the GS scale. This can include financial regulators (higher pay to avoid revolving door problems) as well as "cybersecurity" billets across different places.

rantwasp 7y ago

this story is so familiar that it’s not even funny. I also have seen this while the project is developed. No incentives for fixing stuff that would exponentially increase the productivity of everyone - just keep adding shit until you move on. Pretend like learning the obscurities of a home-brewed/hastily hacked mess matter.

naringas 7y ago
a few years of that until one day, for example Apple realeases Catalina?

gwbas1c 7y ago

> I suspect this permanently stunted my professional growth, but this was during the great recession and I took whatever work I could find From reading your post, I think this was excellent experience in understanding office politics; and in software development. One of the ways to learn how to write maintainable software is to maintain someone else's software!

kaycebasques 7y ago
I’m also curious why OP should think that this might permanently stunt a career.
- non-entity 7y ago
  OP mentioned the work was on legacy systems, and many of the top jobs that everyone want are likely going to be doing new development with new(ish) tech. I'd imagine especially doing legacy work at a non-tech company will not look very good on your resume to a tech company (whether that be a startup, small software firm, or big n) Just my guess
  - jcadam 7y ago
    > ... doing legacy work at a non-tech company will not look very good on your resume to a tech company... This, more or less. Maintenance also tends to be a very low-visibility activity at most companies as well. Promotions go to folks who can make a name for themselves working on the company's hot new project. It took a while to find a job using modern tech when I had a resume full of things like Solaris 8, Ada95, ClearCase, and Oracle. I finally found a company to give me a chance doing new development using modern tech (well, Java, at least) back in 2012 (in my early thirties). My career has been doing much better since, but I feel like I'm far behind where I should be for someone my age (not in skill, mind you, but in pay, role, status, etc).

FoomFries 7y ago

I find the trick is to do maintenance while doing meaningful (as described by the powers that be) work. So when a new feature or bug fix is asked for, add another 15% to the time required, and clean up the code base a bit. The knee-jerk, which rarely works, is to demand time to refactor everything or ignore everything until it breaks irreparably. I think in your case, simplifying a script by a dozen lines a week or so could have been done without taking away "points". Abandoning the status quo and committing too much time will inevitably be looked down on in most industries, however.

bluGill 7y ago
The poster was doing a military contract, not working for his own company. His companies money - and thus his bonus - is in putting in hours to do what is requested. Saving money means they get paid less hours now the project is smaller and so the bonus is less because they are paid less. In private industry it can (but often does not) mean that you can get more profit and thus a larger bonus by saving your company money long term. However in the world of contractors saving the client money this is not rewarded.

natalyarostova 7y ago

> permanently stunted my professional growth This is such a counter-productive attitude.