5 ms·

cloud computings darker secret is armies of overworked sysadmins and SRE's toiling in obscurity through exhausting oncall hours behind the scenes to keep everyt

cloud computings darker secret is armies of overworked sysadmins and SRE's toiling in obscurity through exhausting oncall hours behind the scenes to keep everything running despite these sorts of cuts and mountains of tech debt.. anyone whos worked for a hosting provider can attest.

spenczar5 4y ago
Is that really a secret? I have worked both on the inside at AWS, and on the outside at groups which used AWS and Google Cloud. “They will do the huge amount of sysadmin toil” is, like, the number one selling point versus using on-prem systems, and its what justifies the prices.
- throwboatyface 4y ago
  The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.
  - wnolens 4y ago
    It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(
  - markus_zhang 4y ago
    I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.
    - yourapostasy 4y ago
      > I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.
  - tempnow987 4y ago
    They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.
  - lightbendover 4y ago
    All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.
  - Aperocky 4y ago
    But it's not BS, automation is essential. I'd look for other reasons..
  - twblalock 4y ago
    And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.
- pclmulqdq 4y ago
  The amount of sysadmin/SRE toil in some parts of the stack (all the distributed computing coordination layers) is super-linear in the growth of the system, but the nuanced version of the claim is that they can make up for it in the other parts of the stack (hardware cost and better global availability). A lot of people who have never seen these systems from the inside would be surprised at (1) how much engineering work is put into just getting things like "Postgres but it scales" and "Kubernetes but it works on our proprietary VM manager" to work and (2) how many internal incidents these cloud providers have that are quietly managed before they rise to the level where a customer would notice or care.
hinkley 4y ago
The “best” thing about layoffs/attrition is that onerous task you used to split with several people you now have to do twice as often.
zarathustreal 4y ago
(opinions my own) - I actually do work for $FAANG and can confirm this statement. Mountains of tech debt, 20+ year old tech running most of the stack, most tooling written in dynamic interpreted languages, and above all else every single engineer is expected to be part of on-call rotations, even if your role is not related to infrastructure or operations whatsoever. Even if you don't touch the service(s) your team maintains because you're in a FEE role. The SDE role at $FAANG I work at is essentially glorified tech support, which makes the leetcode interviewing process so much more ridiculous than it would be by default if you were actually writing code and solving meaningful problems most of the time.
- shric 4y ago
  > The SDE role at (FAANG) I don't think you're generalising to all FAANG, just whichever one you work at. That said, this has not been my experience at all at another FAANG which doesn't use the TLA "SDE".
  - zarathustreal 4y ago
    Using (FAANG) for anonymity, but you're absolutely correct
  - influx 4y ago
    I’ve worked at Twitter, Uber, Amazon, and Facebook. They all had mountains of tech debt and oncall burden. Which one do you work at so I can go there?
    - ryandrake 4y ago
      I think you're going to find this at nearly all software companies, FAANG or smaller. The sad state of our industry is that everything is so poorly written that it's falling apart daily, and each company has an army of people trying to toss water on the last fire that popped up. This is the natural steady state of an industry that prioritizes schedule and feature cram over quality and craftsmanship, and prioritizes human heroics/toil over process and automation. Move fast and break things.
walrus01 4y ago
It's well known in the ISP industry that being an entry to mid level sysadmin/network eng at a hosting provider that has large numbers of very-low-dollar-amount per month customers is a new circle of personal hell. Working endless tickets from people who pay $20 a month for some VPS. The only way to get out of it is to become senior enough to not care anymore or work for a hosting operator that specializes in only higher dollar value colo/dedicated server customers.
- deleted 4y ago
  [deleted]
paxys 4y ago
What's the "dark secret" exactly? The entire point of AWS is companies being able to outsource this army of sysadmins so they don't have to hire them themselves. That's also why they pay 3-5x the cost of buying an equivalent server to Amazon.
vishnugupta 4y ago
From multiple reliable sources I've heard that AWS's DynanamoDB team/ops is a meat grinder. The open tickets run into thousands with no hope of anyone getting time to work on permanent fixes.

spenczar5 4y ago

Is that really a secret? I have worked both on the inside at AWS, and on the outside at groups which used AWS and Google Cloud. “They will do the huge amount of sysadmin toil” is, like, the number one selling point versus using on-prem systems, and its what justifies the prices.

throwboatyface 4y ago
The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.
- wnolens 4y ago
  It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(
- markus_zhang 4y ago
  I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.
  - yourapostasy 4y ago
    > I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.
- tempnow987 4y ago
  They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.
- lightbendover 4y ago
  All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.
- Aperocky 4y ago
  But it's not BS, automation is essential. I'd look for other reasons..
- twblalock 4y ago
  And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.
pclmulqdq 4y ago
The amount of sysadmin/SRE toil in some parts of the stack (all the distributed computing coordination layers) is super-linear in the growth of the system, but the nuanced version of the claim is that they can make up for it in the other parts of the stack (hardware cost and better global availability). A lot of people who have never seen these systems from the inside would be surprised at (1) how much engineering work is put into just getting things like "Postgres but it scales" and "Kubernetes but it works on our proprietary VM manager" to work and (2) how many internal incidents these cloud providers have that are quietly managed before they rise to the level where a customer would notice or care.

hinkley 4y ago

The “best” thing about layoffs/attrition is that onerous task you used to split with several people you now have to do twice as often.

zarathustreal 4y ago

(opinions my own) - I actually do work for $FAANG and can confirm this statement. Mountains of tech debt, 20+ year old tech running most of the stack, most tooling written in dynamic interpreted languages, and above all else every single engineer is expected to be part of on-call rotations, even if your role is not related to infrastructure or operations whatsoever. Even if you don't touch the service(s) your team maintains because you're in a FEE role. The SDE role at $FAANG I work at is essentially glorified tech support, which makes the leetcode interviewing process so much more ridiculous than it would be by default if you were actually writing code and solving meaningful problems most of the time.

shric 4y ago
> The SDE role at (FAANG) I don't think you're generalising to all FAANG, just whichever one you work at. That said, this has not been my experience at all at another FAANG which doesn't use the TLA "SDE".
- zarathustreal 4y ago
  Using (FAANG) for anonymity, but you're absolutely correct
- influx 4y ago
  I’ve worked at Twitter, Uber, Amazon, and Facebook. They all had mountains of tech debt and oncall burden. Which one do you work at so I can go there?
  - ryandrake 4y ago
    I think you're going to find this at nearly all software companies, FAANG or smaller. The sad state of our industry is that everything is so poorly written that it's falling apart daily, and each company has an army of people trying to toss water on the last fire that popped up. This is the natural steady state of an industry that prioritizes schedule and feature cram over quality and craftsmanship, and prioritizes human heroics/toil over process and automation. Move fast and break things.

walrus01 4y ago

It's well known in the ISP industry that being an entry to mid level sysadmin/network eng at a hosting provider that has large numbers of very-low-dollar-amount per month customers is a new circle of personal hell. Working endless tickets from people who pay $20 a month for some VPS. The only way to get out of it is to become senior enough to not care anymore or work for a hosting operator that specializes in only higher dollar value colo/dedicated server customers.

deleted 4y ago
[deleted]

paxys 4y ago

What's the "dark secret" exactly? The entire point of AWS is companies being able to outsource this army of sysadmins so they don't have to hire them themselves. That's also why they pay 3-5x the cost of buying an equivalent server to Amazon.

vishnugupta 4y ago

From multiple reliable sources I've heard that AWS's DynanamoDB team/ops is a meat grinder. The open tickets run into thousands with no hope of anyone getting time to work on permanent fixes.