9 ms·

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in the

I can't help but wonder, with the increases in attrition across the industry, are we hitting some kind of tipping point where the institutional knowledge in these massive tech corporations is disappearing? Mistakes happen all the time but when all the people who intimately know how these systems work leave for other opportunities, disasters are bound to happen more and more.

zwirbl 5y ago
Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running
- viraptor 5y ago
  Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?
  - aaronax 5y ago
    Or how about Anathem, with the Ita class doing computer things and nuclear materials cared for by a select group?
    - 9wzYQbTYsAIc 5y ago
      That was probably one of Stephenson’s most insightful moments of science fiction. Scientists only get to talk to the public every 100 years or something, wasn’t it? IT was never allowed to talk to scientists. Seemed like a modernist idea, even at the time of publishing.
  - atty 5y ago
    Not far off. The “golden age” of humanity was shattered long ago, with the mortal wounding of the god emperor, and knowledge of most of the greatest technology was lost.Millennia later, a cult has grown up that both worships and maintains technology as having machine spirits, which are somehow linked to the machine god itself. That god may or may not be the same or related to the god emperor of mankind, depending on the interpretation. Honestly the lore of w40k is quite fun to read, if you’re into dystopian and fantasy sci-fi.
    - R0b0t1 5y ago
      FWIW hardline tech priests view the machine god as separate from the emperor. Hardline imperium officials view the emperor as separate from the machine god. The official party line is that the emperor and the machine god are the same, with the emperor perhaps being an avatar. It seems like both sides are fine to have them be reconciled, but it's an important narrative gadget that can be used to get humanity to fight itself in-universe. Also interesting is that humanity's "lost" technological progress seems to eclipse some of the other races in the W40K narrative, with even the Tau (space dwarves with robots) and Eldar (space elves with crystals) freaking out when humanity brings giant robots, because the sheer physical impracticality of a gigantic human shaped robot is noted, with nobody aware of how they continue to work.
      - 9wzYQbTYsAIc 5y ago
        > gigantic human shaped robot is noted, with nobody aware of how they continue to work BattleTech had something similar: it was considered cheaper to keep replacing humans than to replace the mechs, because hardly anyone knew how to repair or build mechs.
- SketchySeaBeast 5y ago
  If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it. "What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."
  - rurp 5y ago
    Nothing makes me feel more like a wizard than learning some new command line skills. Just string together some short, seemingly indecipherable symbols, and... magic!
  - deleted 5y ago
    [deleted]
  - Mezzie 5y ago
    As an archivist/librarian-programmer, I'm totally here for calling myself a tech priestess.
    - 9wzYQbTYsAIc 5y ago
      Cold
  - anonporridge 5y ago
    Walk down the rows of cubicles chanting "Nonne avertis et conare iterum?" (best I could translate "have you tried turning it off and on again" to latin)
- hughrr 5y ago
  So today I find out my job title is tech priest. I was happy with necromancer before. Does it come with a pay rise?
  - hinkley 5y ago
    Not at all, but a status increase for sure.
fragmede 5y ago
You're right, but that's been true since the beginning of the tech boom (but isn't exclusive to tech) when no one works for a place for several decades. Companies weather this in different ways but attrition has always been around. What's causing people to believe that the latest round of attrition is any different?
- hkt 5y ago
  I'd speculate that perhaps more senior people are moving, and/or a greater overall rate of attrition combined with much more complex technologies and organisations. In other words, it might be harder to become good at jobs now, and fewer people stick with them. Just a hunch but definitely seems to be where the incentives point with loyalty penalties and tech bloat.
  - 9wzYQbTYsAIc 5y ago
    In my experience, education certainly seems to not have kept up with computing, at least in terms of having massive academic-industry partnerships like a doctors residency or a trade apprenticeship . I’d definitely agree that it is probably harder to become good in older organizations - the technologies are probably generations behind the current state of the art and the learning curves are high for those older technologies. Just thinking through keyboard, but it’s probably reaching the point where enterprises need to evaluate aqui-hire or outsourcing entire development departments due to attrition, due to the incentives to leave for regular employees.
- 9wzYQbTYsAIc 5y ago
  > What's causing people to believe that the latest round of attrition is any different? The Great Recession The Great Resignation The Great Dying (due to COVID-19)
  - spoils19 5y ago
    > The Great Dying (due to COVID-19) Repeating this wise comment: https://news.ycombinator.com/item?id=23769427 https://news.ycombinator.com/item?id=23769427 The COVID death counts are hopelessly over-counted. This is why there's a cottage industry of people pointing out things like "COVID deaths" which mysteriously also suffered from being murdered, or drug overdoses, or undiagnosed leukaemia. Then you get into the problem of care homes being authorised to report COVID deaths without any testing or formally trained opinion at all.
nyellin 5y ago
That's the problem we're out to solve with robusta.dev. We're slowly but surely converting the world's institutional technical knowledge into re-usable and automated runbooks.
- hughrr 5y ago
  I’m just going to have to spend all day fixing the runbooks as well as the technology ;)
  - nyellin 5y ago
    I could make the same argument for why dockerfiles are bad and you should install infrastructure manually instead of automating it!
Traster 5y ago
I think this is a transient issue. When you're in growth mode you make a huge series of hacks to just keep things running and then when you leave.... well, it's a problem. But if the business is robust, and lives beyond you, what replaces your work is better documented, better tested, and maintainable. That's the dream. Obviously there are companies that sink between v1 and v2, but that's life. Fundamentally I think the cloud business is robust, it's a fundamentally reasonable way of organising things (for enough people), which is why it attracts customers despite being arguably more expensive. I've been in this situation in much smaller scales, and yes, you'll see massive drop in productivity but that's the cost of going from prototype to product.
- replygirl 5y ago
  if us-east-1 is "in growth mode", what that we rely on can we possibly expect to ever reach maturity?
  - Traster 5y ago
    No what I'm saying is that AWS is transitioning out of early stage growth, and so they're seeing all the issues you see where the original people who hacked stuff together are moving on and you have to start really focusing on making things stable, but the beginning of that process is always everything getting unreliable.
thethethethe 5y ago
Disc: Googler opinions are my own. I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined. It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent outages if things are setup correctly. In theory, my dog should be able to sit on my laptop and submit a prod breaking change without any fear that it will make it to prod and damage the system because automated testing/canary should catch it and, if it does make it to prod, we should be able to detect and mitigate it before the change effects more users using probes or whitebox monitoring. This is happens for 99.9% of potential issues and is completely invisible to users. However it's what's not caught (the remaining 0.01%) that actually matters.
- dijit 5y ago
  I’ll be honest. My external impression of Amazon and Google could not be more distant in this regard. Google doesn’t have nearly as hard a time retaining good people as Amazon does.
  - AitchEmArsey 5y ago
    From within AWS, it just feels like we push people too hard. Service teams are too small relative to their goals, sales teams have unrealistic growth targets and also double as support for needy/incompetent customers, and professional services have billable hour requirements which are as high as any major consultancy and with additional pre-sales support expectations. Strict adherence to the "hiring bar" means we fail to bring in good people who aren't desperate enough to act out the cultish LP dance during their interview. Hiring new grads seems to be the only area where growth is not stalling - but that can obviously only help so much. My team is hiring for 2-3 people and we are being buried alive without that growth happening sooner - but I can't in good conscience recommend this place to anyone I respect or like.
    - kodah 5y ago
9wzYQbTYsAIc 5y ago
Outside of the mega-fang industry, I’m wondering the same thing. The Great Resignation had to have taken a huge toll on regular enterprises. There are probably going to be some unlucky (or lucky, depending on how hardcore they are) people in the position of maintaining aging legacy systems and retrofitting them into the future. COBOL, for example, is becoming a lucrative language for people in the financial and insurance industries. Legacy Java is all over the place, I’m sure. Legacy .NET is in the middle of a huge industry retrofit, (.NET 5 was the official post-legacy rebrand and they’re on to .NET 6+ now).
- Lascaille 5y ago
  >The Great Resignation had to have taken a huge toll on regular enterprises. The Great Resignation was people leaving jobs they didn't want (front of house/service industry/gig) for jobs they did want (career-track jobs) after those jobs resumed hiring again after the pandemic settled down. Labor force participation went up not down due to 'the great resignation.'
  - 9wzYQbTYsAIc 5y ago
    We were discussing attrition, not labor force participation.
newobj 5y ago
No
- 9wzYQbTYsAIc 5y ago
  That’s good to hear. How wide is your scope of experience?
la64710 5y ago
Resolved in 7 mins. Can you do better?
- 9wzYQbTYsAIc 5y ago
  Nice work, keep it up. It’s surely not easy managing that many companies computer systems.
- iso1210 5y ago
  My monitoring doesn't remember the last time we had a service outage lasting 7 seconds, let alone 7 minutes.
la64710 5y ago
faangiq 5y ago
Yep. They literally need to start doubling pay to retain people. The attrition this year is going to be devastating.
steveBK123 5y ago
Yes, absolutely. Within my own org of ~50 people, 15% have resigned/contracts ending during Q1 (after 15% in Q4). Of the remaining 85%.. 20% have been around since before COVID / 65% joined during COVID. Of our senior engineers & team leads, 70% have joined in last 6-9 months. Only 3 full time senior engineers with 2 years or more tenure. We've grown during COVID but we've also just burned through people. Turnover has hit the point where we stopped doing going away zoom toasts.. people just sort of disappeared.
- 9wzYQbTYsAIc 5y ago
  Similar story in smaller orgs, from what I’ve heard - people don’t even bother with the going-away stuff any more.
  - metadat 5y ago
    It seems like that is more of an artifact of the pandemic and everyone working remote. Are you going to hold a farewell zoom party? I don't think so.
    - 9wzYQbTYsAIc 5y ago
      > Are you going to hold a farewell zoom party? I would, and have. Granted not with an entire org, but it’s still good form to take a few moments to say goodbye.
      - steveBK123 5y ago
        Yup, it's not that hard to be decent. If we can have meeting after meeting for working groups, agile kayfabe, status reports, etc for hours recurring weekly.. we can spend 15min on the phone saying thanks, good luck, and see you again a handful of times per year when a teammate leaves.
    - steveBK123 5y ago
      Decent people who respect each other as former/possibly future teammates can & do.

zwirbl 5y ago

Just like the tech priests in Warhammer 40k, keeping occult old engineering, thatno one could build anymore, running

viraptor 5y ago
Not familiar with 40k. Was it a similar idea to nuclear-power-as-religion from Foundation?
- aaronax 5y ago
  Or how about Anathem, with the Ita class doing computer things and nuclear materials cared for by a select group?
  - 9wzYQbTYsAIc 5y ago
    That was probably one of Stephenson’s most insightful moments of science fiction. Scientists only get to talk to the public every 100 years or something, wasn’t it? IT was never allowed to talk to scientists. Seemed like a modernist idea, even at the time of publishing.
- atty 5y ago
  Not far off. The “golden age” of humanity was shattered long ago, with the mortal wounding of the god emperor, and knowledge of most of the greatest technology was lost.Millennia later, a cult has grown up that both worships and maintains technology as having machine spirits, which are somehow linked to the machine god itself. That god may or may not be the same or related to the god emperor of mankind, depending on the interpretation. Honestly the lore of w40k is quite fun to read, if you’re into dystopian and fantasy sci-fi.
  - R0b0t1 5y ago
    FWIW hardline tech priests view the machine god as separate from the emperor. Hardline imperium officials view the emperor as separate from the machine god. The official party line is that the emperor and the machine god are the same, with the emperor perhaps being an avatar. It seems like both sides are fine to have them be reconciled, but it's an important narrative gadget that can be used to get humanity to fight itself in-universe. Also interesting is that humanity's "lost" technological progress seems to eclipse some of the other races in the W40K narrative, with even the Tau (space dwarves with robots) and Eldar (space elves with crystals) freaking out when humanity brings giant robots, because the sheer physical impracticality of a gigantic human shaped robot is noted, with nobody aware of how they continue to work.
    - 9wzYQbTYsAIc 5y ago
      > gigantic human shaped robot is noted, with nobody aware of how they continue to work BattleTech had something similar: it was considered cheaper to keep replacing humans than to replace the mechs, because hardly anyone knew how to repair or build mechs.
SketchySeaBeast 5y ago
If we want to normalize letting long term support people call themselves tech priests I'd very much appreciate it. "What were your duties at your last position?" "Performing the daily ministrations and singing the praise of the machine god."
- rurp 5y ago
  Nothing makes me feel more like a wizard than learning some new command line skills. Just string together some short, seemingly indecipherable symbols, and... magic!
- deleted 5y ago
  [deleted]
- Mezzie 5y ago
  As an archivist/librarian-programmer, I'm totally here for calling myself a tech priestess.
  - 9wzYQbTYsAIc 5y ago
    Cold
- anonporridge 5y ago
  Walk down the rows of cubicles chanting "Nonne avertis et conare iterum?" (best I could translate "have you tried turning it off and on again" to latin)
hughrr 5y ago
So today I find out my job title is tech priest. I was happy with necromancer before. Does it come with a pay rise?
- hinkley 5y ago
  Not at all, but a status increase for sure.

fragmede 5y ago

You're right, but that's been true since the beginning of the tech boom (but isn't exclusive to tech) when no one works for a place for several decades. Companies weather this in different ways but attrition has always been around. What's causing people to believe that the latest round of attrition is any different?

hkt 5y ago
I'd speculate that perhaps more senior people are moving, and/or a greater overall rate of attrition combined with much more complex technologies and organisations. In other words, it might be harder to become good at jobs now, and fewer people stick with them. Just a hunch but definitely seems to be where the incentives point with loyalty penalties and tech bloat.
- 9wzYQbTYsAIc 5y ago
  In my experience, education certainly seems to not have kept up with computing, at least in terms of having massive academic-industry partnerships like a doctors residency or a trade apprenticeship . I’d definitely agree that it is probably harder to become good in older organizations - the technologies are probably generations behind the current state of the art and the learning curves are high for those older technologies. Just thinking through keyboard, but it’s probably reaching the point where enterprises need to evaluate aqui-hire or outsourcing entire development departments due to attrition, due to the incentives to leave for regular employees.
9wzYQbTYsAIc 5y ago
> What's causing people to believe that the latest round of attrition is any different? The Great Recession The Great Resignation The Great Dying (due to COVID-19)
- spoils19 5y ago
  > The Great Dying (due to COVID-19) Repeating this wise comment: https://news.ycombinator.com/item?id=23769427 https://news.ycombinator.com/item?id=23769427 The COVID death counts are hopelessly over-counted. This is why there's a cottage industry of people pointing out things like "COVID deaths" which mysteriously also suffered from being murdered, or drug overdoses, or undiagnosed leukaemia. Then you get into the problem of care homes being authorised to report COVID deaths without any testing or formally trained opinion at all.

nyellin 5y ago

That's the problem we're out to solve with robusta.dev. We're slowly but surely converting the world's institutional technical knowledge into re-usable and automated runbooks.

hughrr 5y ago
I’m just going to have to spend all day fixing the runbooks as well as the technology ;)
- nyellin 5y ago
  I could make the same argument for why dockerfiles are bad and you should install infrastructure manually instead of automating it!

Traster 5y ago

I think this is a transient issue. When you're in growth mode you make a huge series of hacks to just keep things running and then when you leave.... well, it's a problem. But if the business is robust, and lives beyond you, what replaces your work is better documented, better tested, and maintainable. That's the dream. Obviously there are companies that sink between v1 and v2, but that's life. Fundamentally I think the cloud business is robust, it's a fundamentally reasonable way of organising things (for enough people), which is why it attracts customers despite being arguably more expensive. I've been in this situation in much smaller scales, and yes, you'll see massive drop in productivity but that's the cost of going from prototype to product.

replygirl 5y ago
if us-east-1 is "in growth mode", what that we rely on can we possibly expect to ever reach maturity?
- Traster 5y ago
  No what I'm saying is that AWS is transitioning out of early stage growth, and so they're seeing all the issues you see where the original people who hacked stuff together are moving on and you have to start really focusing on making things stable, but the beginning of that process is always everything getting unreliable.

thethethethe 5y ago

Disc: Googler opinions are my own. I've been an SRE on a tier 1 GCP product for over three years this is not the case. In my experience, our systems have only gotten more reliable and easier to understand since I joined. It's not like there are only a few key knowledge holders that single handedly prevent outages from happening. In reality, you don't need to know shit about how a system works internally to prevent outages if things are setup correctly. In theory, my dog should be able to sit on my laptop and submit a prod breaking change without any fear that it will make it to prod and damage the system because automated testing/canary should catch it and, if it does make it to prod, we should be able to detect and mitigate it before the change effects more users using probes or whitebox monitoring. This is happens for 99.9% of potential issues and is completely invisible to users. However it's what's not caught (the remaining 0.01%) that actually matters.

dijit 5y ago
I’ll be honest. My external impression of Amazon and Google could not be more distant in this regard. Google doesn’t have nearly as hard a time retaining good people as Amazon does.
- AitchEmArsey 5y ago
  From within AWS, it just feels like we push people too hard. Service teams are too small relative to their goals, sales teams have unrealistic growth targets and also double as support for needy/incompetent customers, and professional services have billable hour requirements which are as high as any major consultancy and with additional pre-sales support expectations. Strict adherence to the "hiring bar" means we fail to bring in good people who aren't desperate enough to act out the cultish LP dance during their interview. Hiring new grads seems to be the only area where growth is not stalling - but that can obviously only help so much. My team is hiring for 2-3 people and we are being buried alive without that growth happening sooner - but I can't in good conscience recommend this place to anyone I respect or like.
  - kodah 5y ago

9wzYQbTYsAIc 5y ago

Outside of the mega-fang industry, I’m wondering the same thing. The Great Resignation had to have taken a huge toll on regular enterprises. There are probably going to be some unlucky (or lucky, depending on how hardcore they are) people in the position of maintaining aging legacy systems and retrofitting them into the future. COBOL, for example, is becoming a lucrative language for people in the financial and insurance industries. Legacy Java is all over the place, I’m sure. Legacy .NET is in the middle of a huge industry retrofit, (.NET 5 was the official post-legacy rebrand and they’re on to .NET 6+ now).

Lascaille 5y ago
>The Great Resignation had to have taken a huge toll on regular enterprises. The Great Resignation was people leaving jobs they didn't want (front of house/service industry/gig) for jobs they did want (career-track jobs) after those jobs resumed hiring again after the pandemic settled down. Labor force participation went up not down due to 'the great resignation.'
- 9wzYQbTYsAIc 5y ago
  We were discussing attrition, not labor force participation.

newobj 5y ago

9wzYQbTYsAIc 5y ago
That’s good to hear. How wide is your scope of experience?

la64710 5y ago

Resolved in 7 mins. Can you do better?

9wzYQbTYsAIc 5y ago
Nice work, keep it up. It’s surely not easy managing that many companies computer systems.
iso1210 5y ago
My monitoring doesn't remember the last time we had a service outage lasting 7 seconds, let alone 7 minutes.

la64710 5y ago

faangiq 5y ago

Yep. They literally need to start doubling pay to retain people. The attrition this year is going to be devastating.

steveBK123 5y ago

Yes, absolutely. Within my own org of ~50 people, 15% have resigned/contracts ending during Q1 (after 15% in Q4). Of the remaining 85%.. 20% have been around since before COVID / 65% joined during COVID. Of our senior engineers & team leads, 70% have joined in last 6-9 months. Only 3 full time senior engineers with 2 years or more tenure. We've grown during COVID but we've also just burned through people. Turnover has hit the point where we stopped doing going away zoom toasts.. people just sort of disappeared.

9wzYQbTYsAIc 5y ago
Similar story in smaller orgs, from what I’ve heard - people don’t even bother with the going-away stuff any more.
- metadat 5y ago
  It seems like that is more of an artifact of the pandemic and everyone working remote. Are you going to hold a farewell zoom party? I don't think so.
  - 9wzYQbTYsAIc 5y ago
    > Are you going to hold a farewell zoom party? I would, and have. Granted not with an entire org, but it’s still good form to take a few moments to say goodbye.
    - steveBK123 5y ago
      Yup, it's not that hard to be decent. If we can have meeting after meeting for working groups, agile kayfabe, status reports, etc for hours recurring weekly.. we can spend 15min on the phone saying thanks, good luck, and see you again a handful of times per year when a teammate leaves.
  - steveBK123 5y ago
    Decent people who respect each other as former/possibly future teammates can & do.