4 ms·
They also saved the salaries of the team whose job was doing nothing but chasing misplaced spaces in yaml configuration files. Cloud infrastructure doesn't just
by awslol 3y ago
They also saved the salaries of the team whose job was doing nothing but chasing misplaced spaces in yaml configuration files. Cloud infrastructure doesn't just appear out of thin air. You have to hire people to describe what you want to do. And with the complexity mess we're in today it's not at all clear which takes more effort.
- icedchai 3y agoTo manage a large fleet of physical servers, you need similar ops skills. You're not going to configure all those systems by hand, are you?
- awslol 3y agoDepends on the size of the fleet. If you're using less than a dozen servers manual configuration is simpler. Depending on what you're doing that could mean serving a hundred million customers. Which is plenty for most business.
- icedchai 3y agoA dozen servers would be pushing it. It's not the size of the fleet, it's the consistency and repeatability of configuration and deployment. I assume there are other, non-production servers: various dev environments, test/QA. How do you ensure every environment is built to spec without some level of automation? It's the old "pets" vs "cattle" argument. I've worked at companies with their own data centers and manual configuration. Every system was a pet.
- postalrat 3y agoThey spent $150,000 on physical servers. Probably 1 or 2 racks. Not much of a 'fleet'.
- ketzo 3y agoI mean, “fleet” semantics aside, surely $150,000 of servers is enough for at least one full-time person to be maintaining them. The point is that there are absolutely maintenance and ops costs associated with these servers.
- dijit 3y agoThere's maintenance cost to everything that runs a userland. What we're missing is tracking what time is spent managing hardware and firmware, (or even network config, if we're being generous) and how much time is being spent on OS config. From personal experience (as a sysadmin before it was entirely unsexy as a term) the overwhelming majority of my ops work was done in userland on the machine, maybe something like 96-97% of my tasks were nothing to do with hardware at all. Since I got rebranded as SRE, the tools and the pay sure did get a lot better in the time, but the job is largely similar and ultimately running in VMs does make deployment faster, but once deployed I find the maintenance burden to be the same (or perhaps a little more) as things seem to become deprecated or require changes from our cloud vendor a bit more often.
- quickthrower2 3y ago100% this. Cloud is a hard slog too. A different slog though. We spend a lot of time chasing Azure deprecations. They are closing down a type of MySQL instance for example for one which is more “modern” but from the end user point of view it is still a MySQL server!
- gymbeaux 3y agoExactly. Last job I worked at there was always an issue with the YAML… and as a “mere” software engineer, I had to wait for offshore DevOps to fix, but that’s another issue.
- scarface_74 3y agoIf you are waiting on a “DevOps department”, it isn’t a DevOps…it’s operations
- gymbeaux 3y agoMy company called them the DevOps Team
- icedchai 3y agoDid you try and fix it yourself? Was that not allowed?
- gymbeaux 3y agoYeah that was not allowed. We started off managing our own DevOps in our own AWS region even, separate from the rest of the company. Eventually the DevOps team mandated we move to the same region as everyone else, and then soon after that there was enough conflict with my (offshore) team and DevOps (onshore) that DevOps demanded and got complete control over our DevOps, though not infrastructure.
- notnmeyer 3y agosorry, what? theyre running on k8s and using helm... so there's still piles of yaml. its wild to conflate migrating to bare metal with eliminating yaml-centric configuration.
- PartiallyTyped 3y agoCDK exists. Haven’t had to use yaml in ages and I refuse to.