5 ms·

The problem is a bit more insidious than that As an engineer, you want to be working on cool new features too! Very few folks will be content sitting on their

The problem is a bit more insidious than that As an engineer, you want to be working on cool new features too! Very few folks will be content sitting on their laurels just fixing the occasional bug or adding a touch more polish to a product that's already "done" If you setup a team to work that way, very soon you'll find that most of your engineers have left. Heck, the manager might get bored and leave too. "Okay, that's fine," you might think "The product is still doing alright even without an owner. Higher level leadership should be fine with that" Until the day comes when the service crashes unexpectedly, and you realize that no one left on the engineering team has enough context to debug the issue properly Hello two week long outage Examples: Heroku - https://twitter.com/GergelyOrosz/status/1520770263977271296 https://twitter.com/GergelyOrosz/status/1520770263977271296 Atlassian - https://twitter.com/GergelyOrosz/status/1513605414029516806 https://twitter.com/GergelyOrosz/status/1513605414029516806

simion314 4y ago
Isn't the 20% time to work on whatever cool shit you want enough ? (like how googlers created that garbage angular1 because they were border, have no clue about GUIs and had some fun scewing around ) ? I know people that are fine with getting paid to maintain shit so maybe the problem is Google only hires cool developers and the cool developers only want to work on cool stuff and in 2 years the newest cool stuff of-course.
- WalterSear 4y ago
  It's a lie.
angarg12 4y ago
There is that too, but you could solve that by having engineers/teams working in new, cool, and useful products. I feel my company doesn't have a good mechanism to maintain services that aren't actively developed. We experience it first hand when one of our services got deprecated and we moved to a new org. The solution was literally to hire a new team in a low CoL country and hand over the service to them. Needless to say it was difficult to hire for those positions.
alimov 4y ago
Experienced something similar, but it was not an outage. Had several knowledgeable people leave the company, and they all happened to be experts in a particular service. Positions weren’t backfilled even though the people gave advance notice. About two months later we ended up getting out butts kicked when nobody knew the details of the service implementation and the service was expected to be updated to support some new features.. couldn’t get it updated for about 3-4 weeks because we couldn’t afford an outage.
- marcosdumay 4y ago
  > couldn’t get it updated for about 3-4 weeks That doesn't sound like a large issue (but the details can completely change it). Maybe it was the right decision.
- alisonatwork 4y ago
  Arguably those folks who left didn't do quite as good a job as they perhaps should have when they were still there. A high quality developer leaving a service behind should have already written sufficient documentation so that another high quality developer (especially one at the same company) can ramp up more quickly than 3-4 weeks. I think this is just another symptom of many tech companies throwing up their hands and pretending like "legacy" services are inescapable technical debt, when really they just never bothered to emphasize to their employees that services should be built in a maintainable way from the outset.
  - mecha_ghidorah 4y ago
    This is true but only so long as they have been given time to do it. I've left a company, with notice, and they had me working on building stuff almost to my last day. Sure I did have a few "hand over" sessions in my schedule where I was expected to walk other devs through some stuff I was the owner for in a broad sense, but they never gave me time to produce solid docs for anything even knowing I was headed out the door
    - Perseids 4y ago
BlargMcLarg 4y ago
Part of this is due to companies shooting themselves in the foot over and over, recruiting developers looking for challenges rather than grunt developers okay doing largely maintenance for a solid income. If they advocate themselves as providing the challenges for the former and filter out the latter, yes, obviously your employees are going to leave after they have to move that one div by 5 pixels for the umpteenth time and get no mental stimulation for months.
- MivLives 4y ago
  Does anyone recruit grunt devs like that? That honestly sounds like what I'd prefer. I just want to come in, keep the lights on, and have enough mental energy for other stuff after work.
  - lordnacho 4y ago
    I could see a market for that guy, and he takes 10 jobs like that Reddit thread.
  - rileyphone 4y ago
    All across corporate America, there are devs who make 80-120k a year and keep it 9-5 (but really like 11-2). Especially with the recent turmoil in the market, you can find a very easy yet well paying (for normal people) job. Now if only HR would advertise the jobs this way...
  - Firmwarrior 4y ago
    I don't know too much about Google, but I can talk about other companies. Everyone recruits grunt devs like that. That's what every job in Silicon Valley is. It's just that some companies and teams want you to spin it like you're some amazing lone wolf 100x genius while you're scraping dogshit off the bottom of the company's shoes. A big part of it is that the executives decide what's making money for the company, and they'll focus on that. If you're scraping turds on the "rockstar" team in the "rockstar" org, you'll get showered with bonus money and RSUs. If you're scraping turds on a product that none of the VPs care about, you'll probably get screwed over. Some of the people in the non-"ninja"/"wizard"/"rockstar" orgs will do OK because they look like indispensable geniuses, and I think that's what a lot of this sentiment comes from.
somethoughts 4y ago
I think you nailed the crux of the problem. The challenge for engineering management is how to provide metrics to measure your bus factor reduction efforts and the strength of your insurance prior to the emergency. It is highly possible though that the new support team members are actually coasting up until the disaster so you didn't really have the insurance you thought you were paying for.
hibikir 4y ago
You can find maintenance experts too. When someone tells me that they have this important, dangerous, buggy system wthat has trouble handling its ever growing load, and they need someone to come in and fix it, I cannot be any happier. But I know the system has to be really important, and that the fire has to be so bad that upper management is willing to spend the money to let some specialists come in, few questions asked, knowing that they will be rewarded as if they were building a new shiny, high visibility doodad. What is difficult is to have management that will keep that level of attention, instead of realizing that the reason that nobody has heard a bad thing from the system in months, if not years, is because there's a lot of work being done trying to make the system invisible. What usually happens is that after a year or two they forget, and the maintenance programmer jumps to a different fire, typically in a different company, because there is far less reward to keep improving the system than to stop it from being a raging fire.
cmrdporcupine 4y ago
Yeah and this problem is exacerbated at Google by the relatively easy process of moving between teams. Other companies I was at made this hard. Google encourages it. When friends complain to me about Google killing projects, they act as if it's upper management making a decision to kill. And that is sometimes the case. But often it's just: nobody wanted to work on it anymore. And Google isn't the kind of "command and control" culture where you crack the whip and tell all your engineers that they're doing X and assign them. At Google they'll just leave your org and move to some other team, and you won't have the power to stop them. Not saying that's a bad thing, but it has some bad outcomes occasionally.
banannaise 4y ago
There are plenty of people who like fixing the occasional bug and adding a touch more polish. The problem is that this is actively disincentivized on every level. On top of that, a lot of people have internalized this. If the world weren't so obsessed with differentiating pay, you would have people who are enthusiastic about a wider variety of things.

simion314 4y ago

Isn't the 20% time to work on whatever cool shit you want enough ? (like how googlers created that garbage angular1 because they were border, have no clue about GUIs and had some fun scewing around ) ? I know people that are fine with getting paid to maintain shit so maybe the problem is Google only hires cool developers and the cool developers only want to work on cool stuff and in 2 years the newest cool stuff of-course.

WalterSear 4y ago
It's a lie.

angarg12 4y ago

There is that too, but you could solve that by having engineers/teams working in new, cool, and useful products. I feel my company doesn't have a good mechanism to maintain services that aren't actively developed. We experience it first hand when one of our services got deprecated and we moved to a new org. The solution was literally to hire a new team in a low CoL country and hand over the service to them. Needless to say it was difficult to hire for those positions.

alimov 4y ago

Experienced something similar, but it was not an outage. Had several knowledgeable people leave the company, and they all happened to be experts in a particular service. Positions weren’t backfilled even though the people gave advance notice. About two months later we ended up getting out butts kicked when nobody knew the details of the service implementation and the service was expected to be updated to support some new features.. couldn’t get it updated for about 3-4 weeks because we couldn’t afford an outage.

marcosdumay 4y ago
> couldn’t get it updated for about 3-4 weeks That doesn't sound like a large issue (but the details can completely change it). Maybe it was the right decision.
alisonatwork 4y ago
Arguably those folks who left didn't do quite as good a job as they perhaps should have when they were still there. A high quality developer leaving a service behind should have already written sufficient documentation so that another high quality developer (especially one at the same company) can ramp up more quickly than 3-4 weeks. I think this is just another symptom of many tech companies throwing up their hands and pretending like "legacy" services are inescapable technical debt, when really they just never bothered to emphasize to their employees that services should be built in a maintainable way from the outset.
- mecha_ghidorah 4y ago
  This is true but only so long as they have been given time to do it. I've left a company, with notice, and they had me working on building stuff almost to my last day. Sure I did have a few "hand over" sessions in my schedule where I was expected to walk other devs through some stuff I was the owner for in a broad sense, but they never gave me time to produce solid docs for anything even knowing I was headed out the door
  - Perseids 4y ago

BlargMcLarg 4y ago

Part of this is due to companies shooting themselves in the foot over and over, recruiting developers looking for challenges rather than grunt developers okay doing largely maintenance for a solid income. If they advocate themselves as providing the challenges for the former and filter out the latter, yes, obviously your employees are going to leave after they have to move that one div by 5 pixels for the umpteenth time and get no mental stimulation for months.

MivLives 4y ago
Does anyone recruit grunt devs like that? That honestly sounds like what I'd prefer. I just want to come in, keep the lights on, and have enough mental energy for other stuff after work.
- lordnacho 4y ago
  I could see a market for that guy, and he takes 10 jobs like that Reddit thread.
- rileyphone 4y ago
  All across corporate America, there are devs who make 80-120k a year and keep it 9-5 (but really like 11-2). Especially with the recent turmoil in the market, you can find a very easy yet well paying (for normal people) job. Now if only HR would advertise the jobs this way...
- Firmwarrior 4y ago
  I don't know too much about Google, but I can talk about other companies. Everyone recruits grunt devs like that. That's what every job in Silicon Valley is. It's just that some companies and teams want you to spin it like you're some amazing lone wolf 100x genius while you're scraping dogshit off the bottom of the company's shoes. A big part of it is that the executives decide what's making money for the company, and they'll focus on that. If you're scraping turds on the "rockstar" team in the "rockstar" org, you'll get showered with bonus money and RSUs. If you're scraping turds on a product that none of the VPs care about, you'll probably get screwed over. Some of the people in the non-"ninja"/"wizard"/"rockstar" orgs will do OK because they look like indispensable geniuses, and I think that's what a lot of this sentiment comes from.

somethoughts 4y ago

I think you nailed the crux of the problem. The challenge for engineering management is how to provide metrics to measure your bus factor reduction efforts and the strength of your insurance prior to the emergency. It is highly possible though that the new support team members are actually coasting up until the disaster so you didn't really have the insurance you thought you were paying for.

hibikir 4y ago

You can find maintenance experts too. When someone tells me that they have this important, dangerous, buggy system wthat has trouble handling its ever growing load, and they need someone to come in and fix it, I cannot be any happier. But I know the system has to be really important, and that the fire has to be so bad that upper management is willing to spend the money to let some specialists come in, few questions asked, knowing that they will be rewarded as if they were building a new shiny, high visibility doodad. What is difficult is to have management that will keep that level of attention, instead of realizing that the reason that nobody has heard a bad thing from the system in months, if not years, is because there's a lot of work being done trying to make the system invisible. What usually happens is that after a year or two they forget, and the maintenance programmer jumps to a different fire, typically in a different company, because there is far less reward to keep improving the system than to stop it from being a raging fire.

cmrdporcupine 4y ago

Yeah and this problem is exacerbated at Google by the relatively easy process of moving between teams. Other companies I was at made this hard. Google encourages it. When friends complain to me about Google killing projects, they act as if it's upper management making a decision to kill. And that is sometimes the case. But often it's just: nobody wanted to work on it anymore. And Google isn't the kind of "command and control" culture where you crack the whip and tell all your engineers that they're doing X and assign them. At Google they'll just leave your org and move to some other team, and you won't have the power to stop them. Not saying that's a bad thing, but it has some bad outcomes occasionally.

banannaise 4y ago

There are plenty of people who like fixing the occasional bug and adding a touch more polish. The problem is that this is actively disincentivized on every level. On top of that, a lot of people have internalized this. If the world weren't so obsessed with differentiating pay, you would have people who are enthusiastic about a wider variety of things.