4 ms·

As a manager of a team of application product developers I can tell you, the headcount cost of ops teams & the time cost of taking people whose job shouldn’t in

by mlthoughts2018 6y ago

As a manager of a team of application product developers I can tell you, the headcount cost of ops teams & the time cost of taking people whose job shouldn’t involve vm provisioning overhead but nonetheless does are both huge compared to cloud services. In cloud tooling, my team of people all with zero experience doing vm provisioning can get production systems up, add logging, add alerting, add networking, etc., all very easily or with just low touch overhead from teams that manage best practices or compliance. Creating the same internal developer tool experience with data centers is SO expensive and requires a major headcount investment.

AnthonyMouse 6y ago
It's always shocking to see how inefficiently some companies are operated. The things you're describing should take a single individual a matter of seconds for a system which is already in operation, and a one-time cost of a few hours to set up at the outset (i.e. once or twice a decade). If it takes significantly longer to do locally than it takes to input into the cloud provider's interface, something's not right. I can tell you where most companies go wrong here though. It's in excessive specialization. If you put separate people in charge of provisioning, networking, logging, etc. then you create a ton of friction to do anything because you need five different humans to touch it and they all have to coordinate. One person can do all of those things, as you've learned when one person does it interacting with the cloud providers. And when one person is doing all of it, it takes only seconds to do.
- scarface74 6y ago
  I listed all of the services we used across five environments and most across three availability zones. So one person was going to manage what on prem would be roughly 200 VMs/services and make sure they stay patched, the OS stays updated? Locally, a lot of those services would run in a cluster for availability. There is no company on earth that has one person managing an on prem implementation of that level of complexity. One person does it in a cloud environment because they aren’t managing hardware, patching, etc.
  - AnthonyMouse 6y ago
    > So one person was going to manage what on prem would be roughly 200 VMs/services and make sure they stay patched, the OS stays updated? You're talking about the applications, i.e. the guests, not the hosts. It should be completely reasonable to expect a single person to be able to manage the hosts, including the networking for the hosts, the backups of the guest images by the hosts, etc. The services themselves may be arbitrarily complicated and require a large number of people to configure and manage, but that's the same as it is with cloud providers. There is also a simple way to handle guest OS updates in most cases. Turn on automatic updates, and keep automated VM snapshots in case one goes bad.
    - cinquemb 6y ago
      Many devs don't want to do any of that, they would rather have everything run in docker and not worry about any of the above (which if you can afford it at scale it can make sense, personally I'm not of that ethos and try to stay away from companies like that, as its great for me personally to work in a place where the devs can jump in anywhere rather just be tied down in their silo). I think its easier for some companies to just pay amazon for all these services than to hire engineers be able to do what you are talking about. For a certain type of company (well established, known resource loads) it can make sense now (though risk huge lock-in costs and wont be able to move fast enough when they arrive), but I've seen/worked at many start ups that try to do the full cloud setups above and just get blown out from just costs alone. The last company I worked at (south east asia start up, with just me originally as the dev, but grew to about 5 more devs when I left) we only had s3, cloudfront (though we put cdn77 infront of it cause bandwidth in SEA is a rip off with aws), and rds with postgres (and I wanted to move away from this cause backups and resizes from sql dumps are a PITA and take way too long when you have to do it over the network vs on the instance) and ran every other service we needed on ec2 instances (load balancer/auto scaling with nginx+3rd party modules + python daemon with boto3 api, rabbitmq, solr, memcached, dnscache, pgbouncer, letsencryot, deployments to prod/staging/one off envs for random stuff with fabric building what was needed on a workbench then spin up instances from an AMI image [1/4 the cost compared to using docker, with init scripts that turned off/on applications depending on what the server was need for] to about 40+ machines at max site load]) and had to support 40+ customer domains which overall was 8-10x cheaper than the "all-in" AWS abomination an AWS consultant cooked up before we went that route.

AnthonyMouse 6y ago

It's always shocking to see how inefficiently some companies are operated. The things you're describing should take a single individual a matter of seconds for a system which is already in operation, and a one-time cost of a few hours to set up at the outset (i.e. once or twice a decade). If it takes significantly longer to do locally than it takes to input into the cloud provider's interface, something's not right. I can tell you where most companies go wrong here though. It's in excessive specialization. If you put separate people in charge of provisioning, networking, logging, etc. then you create a ton of friction to do anything because you need five different humans to touch it and they all have to coordinate. One person can do all of those things, as you've learned when one person does it interacting with the cloud providers. And when one person is doing all of it, it takes only seconds to do.

scarface74 6y ago
I listed all of the services we used across five environments and most across three availability zones. So one person was going to manage what on prem would be roughly 200 VMs/services and make sure they stay patched, the OS stays updated? Locally, a lot of those services would run in a cluster for availability. There is no company on earth that has one person managing an on prem implementation of that level of complexity. One person does it in a cloud environment because they aren’t managing hardware, patching, etc.
- AnthonyMouse 6y ago
  > So one person was going to manage what on prem would be roughly 200 VMs/services and make sure they stay patched, the OS stays updated? You're talking about the applications, i.e. the guests, not the hosts. It should be completely reasonable to expect a single person to be able to manage the hosts, including the networking for the hosts, the backups of the guest images by the hosts, etc. The services themselves may be arbitrarily complicated and require a large number of people to configure and manage, but that's the same as it is with cloud providers. There is also a simple way to handle guest OS updates in most cases. Turn on automatic updates, and keep automated VM snapshots in case one goes bad.
  - cinquemb 6y ago
    Many devs don't want to do any of that, they would rather have everything run in docker and not worry about any of the above (which if you can afford it at scale it can make sense, personally I'm not of that ethos and try to stay away from companies like that, as its great for me personally to work in a place where the devs can jump in anywhere rather just be tied down in their silo). I think its easier for some companies to just pay amazon for all these services than to hire engineers be able to do what you are talking about. For a certain type of company (well established, known resource loads) it can make sense now (though risk huge lock-in costs and wont be able to move fast enough when they arrive), but I've seen/worked at many start ups that try to do the full cloud setups above and just get blown out from just costs alone. The last company I worked at (south east asia start up, with just me originally as the dev, but grew to about 5 more devs when I left) we only had s3, cloudfront (though we put cdn77 infront of it cause bandwidth in SEA is a rip off with aws), and rds with postgres (and I wanted to move away from this cause backups and resizes from sql dumps are a PITA and take way too long when you have to do it over the network vs on the instance) and ran every other service we needed on ec2 instances (load balancer/auto scaling with nginx+3rd party modules + python daemon with boto3 api, rabbitmq, solr, memcached, dnscache, pgbouncer, letsencryot, deployments to prod/staging/one off envs for random stuff with fabric building what was needed on a workbench then spin up instances from an AMI image [1/4 the cost compared to using docker, with init scripts that turned off/on applications depending on what the server was need for] to about 40+ machines at max site load]) and had to support 40+ customer domains which overall was 8-10x cheaper than the "all-in" AWS abomination an AWS consultant cooked up before we went that route.