5 ms·

The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at al

The cloud providers like to pretend their secret sauce is massive scale automation. They have tooling, alerting, etc. to make this all work with no people at all. Of course it's all BS, and the people I know who came from AWS were some of the worst for throwing bodies at problems and manually tweaking shit without trying to understand or automate solutions.

wnolens 4y ago
It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(
markus_zhang 4y ago
I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.
- yourapostasy 4y ago
  > I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.
tempnow987 4y ago
They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.
lightbendover 4y ago
All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.
Aperocky 4y ago
But it's not BS, automation is essential. I'd look for other reasons..
twblalock 4y ago
And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.

wnolens 4y ago

It is their secret sauce and like a startup who licks and sticks envelopes initially while proving their business model, eventually it is automated. But the thirst for growth is never quenched, so there's always new features, new services. And maintenance of that automation. The work never ends. There's a 25-50% operations tax every AWS employee pays. But they're handling the operations load that several employees were 10y ago. I used to wake up in the middle of the night to restart individual machines. Now I'm alerted for much much more complex problems. But i'm still woken up :(

markus_zhang 4y ago

I admin a small part of our Google cloud part time and I feel I absolutely have no time and no idea how to drill deeper than clickOps. Knowing where is everything and the general risks of toggling a switch are already nerve breaking. Sometimes I wish I could learn everything from scratch. I wish I could join an on prem team to learn all those small details slowly and painfully.

yourapostasy 4y ago
> I wish I could join an on prem team to learn all those small details slowly and painfully. Learning "from the middle" of a tech stack outwards is the default case for everyone, so don't be so hard on yourself. There is no "from scratch", never was even in the Bombe days, just a question of how many abstraction layers you want to tackle. Give yourself time to learn and plenty of room to fail and try again, and "all those small details" will ingrain themselves into you as second nature.

tempnow987 4y ago

They throw bodies because they have them and they are looking to solve a problem. And they do automate, claims that a service like S3 does not have automation are totally ridiculous, and few other general hosting providers have AWS scale here. That said, their development pace is crazy, with plenty of partially built solutions -> so yes, until something scales enough to need it I'm not surprised they have sysadmin toil. Reality is even the VMware worlds and other onprem stuff, even when "automated" have plenty of turmoil just staying fresh / current / backed up / secured etc. So that's what AWS is selling.

lightbendover 4y ago

All of big tech is throwing bodies at problems at this point since decades of data is all rotted and programmatic solutions simply don't work. IF you stay at Amazon after all of this attrition, your life will be hell.

Aperocky 4y ago

But it's not BS, automation is essential. I'd look for other reasons..

twblalock 4y ago

And yet AWS is pretty reliable overall, and certainly better than most in-house systems even at large companies. I know multiple people who have worked on AWS. There is definitely some manual stuff going on, but there is also considerable infrastructure and automation in other areas.