6 ms·

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical

by zosima 2y ago

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

acedTrex 2y ago
I have said for years the value of cloud is mainly its api, thats the selling point in large enterprise.
- sgarland 2y ago
  Self-hosted software also has APIs, and Terraform libraries, and Ansible playbooks, etc. It’s just that you have to know what it is you’re trying to do, instead of asking AWS what collection of XaaS you should use.
sgarland 2y ago
I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everything under the sun, there are Ansible (or Chef, or Salt…) playbooks to do the same. For both, you _should_ know what all of the options are doing. The only way I see this sentiment being true is that a dev team, with no infrastructure experience, can more easily spin up a lot of infra – likely in a sub-optimal fashion – to run their application. When it inevitably breaks, they can then throw money at the problem via vertical scaling, rather than addressing the root cause.
- the__alchemist 2y ago
  Do you need those tools? It seems that for fundamental web hosting, you need your application server, nginx or similar, postgres or similar, and a CLI. (And an interpreter etc if your application is in an interpreted lang)
  - sgarland 2y ago
    I suppose that depends on your RTO. With cloud providers, even on a bare VM, you can to some extent get away with having no IaC, since your data (and therefore config) is almost certainly on networked storage which is redundant by design. If an EC2 fails, or even if one of the drives in your EBS drive fails, it'll probably come back up as it was. If it's your own hardware, if you don't have IaC of some kind – even something as crude as a shell script – then a failure may well mean you need to manually set everything up again.
    - noprocrasted 2y ago
      Get two servers (or three, etc)?
      - sgarland 2y ago
        Well, sure – I was trying to do a comparison in favor of cloud, because the fact that EBS Volumes can magically detach and attach is admittedly a neat trick. You can of course accomplish the same (to a certain scale) with distributed storage systems like Ceph, Longhorn, etc. but then you have to have multiple servers, and if you have multiple servers, you probably also have your application load balanced with failover.
    - champtar 2y ago
      All EBS volumes except io2 have advertised durability of 99.8%, which is pretty low, so don't count it in the magic networked storage category.
  - zbentley 2y ago
    For fundamentals, that list is missing: - Some sort of firewall or network access control. Being able to say "allow http/s from the world (optionally minus some abuser IPs that cause problems), and allow SSH from developers (by IP, key, or both)" at a separate layer from nginx is prudent. Can be ip/tables config on servers or a separate firewall appliance. - Some mechanism of managing storage persistence for the database, e.g. backups, RAID, data files stored on fast network-attached storage, db-level replication. Not losing all user data if you lose the DB server is table stakes. - Something watching external logging or telemetry to let administrators know when errors (e.g. server failures, overload events, spikes in 500s returned) occur. This could be as simple as Pingdom or as involved as automated alerting based on load balancer metrics. Relying on users to report downtime events is not a good approach. - Some sort of CDN, for applications with a frontend component. This isn't required for fundamental web hosting, but for sites with a frontend and even moderate (10s/sec) hit rates, it can become required for cost/performance; CDNs help with egress congestion (and fees, if you're paying for metered bandwidth). - Some means of replacing infrastructure from nothing. If the server catches fire or the hosting provider nukes it, having a way to get back to where you were is important. Written procedures are fine if you can handle long downtime while replacing things, but even for a handful of application components those procedures get pretty lengthy, so you start wishing for automation. - Some mechanism for deploying new code, replacing infrastructure, or migrating data. Again, written procedures are OK, but start to become unwieldy very early on ('stop app, stop postgres, upgrade the postgres version, start postgres, then apply application migrations to ensure compatibility with new version of postgres, then start app--oops, forgot to take a postgres backup/forgot that upgrading postgres would break the replication stream, gotta write that down for net time...'). ...and that's just for a very, very basic web hosting application--one that doesn't need caches, blob stores, the ability to quickly scale out application server or database capacity. Each of those things can be accomplished the traditional way--and you're right, that sometimes that way is easier for a given item in the list (especially if your maintainers have expertise in that item)! But in aggregate, having a cloud provider handle each of those concerns tends to be easier overall and not require nearly as much in-house expertise.
- zosima 2y ago
  You are focusing on technology. And sure of course you can get most of the benefits of AWS a lot cheaper when self-hosting. But when you start factoring internal processes and incompetent IT departments, suddenly that's not actually a viable option in many real-world scenarios.
  - jeffbee 2y ago
    Exactly. With the cloud you can suddenly do all the things your tyrannical Windows IT admin has been saying are impossible for the last 30 years.
    - the_arun 2y ago
      It is similar to cooking at home vs ordering cooked food everyday. If some guarantees the taste & quality people would happy to outsource it.
- tylerchurch 2y ago
  I think this is only true for teams and apps of a certain size. I've worked on plenty of teams with relatively small apps, and the difference between: 1. Cloud: "open up the cloud console and start a VM" 2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc." Is quite large. #1 is 15 minutes for a single team lead. #2 requires the team to agree on hardware specs, get management approval, finance approval, executives signing contracts. And through all this you don't have anything online yet for... weeks? If your team or your app is large, this probably all averages out in favor of #2. But small teams often don't have the bandwidth or the budget.
  - noprocrasted 2y ago
    3. "Dedicated server" at any hosting provider Open their management console, press order now, 15 mins later get your server's IP address.
    - zbentley 2y ago
      For purposes of this discussion, isn't AWS just a very large hosting provider? I.e. most hosting providers give you the option for virtual or dedicated hardware. So does Amazon (metal instances). Like, "cloud" was always an ill-defined term, but in the case of "how do I provision full servers" I think there's no qualitative difference between Amazon and other hosting providers. Quantitative, sure.
      - noprocrasted 2y ago
        > Amazon (metal instances) But you still get nickel & dimed and pay insane costs, including on bandwidth (which is free in most conventional hosting providers, and overages are 90x cheaper than AWS' costs).
      - irunmyownemail 2y ago
        Qualitatively, AWS is greedy and nickle and dime you to death. Their Route53 service doesn't even have all the standard DNS options I need and I can get everywhere else or even on my own running bind9. I do not use IPv6 for several reasons, when AWS decided charge for IPv4, I went looking elsewhere to get my VM's. I can't even imagine how much the US Federal Government is charging American taxpayers to pay AWS for hosting there, it has to be astronomical.
- bonoboTP 2y ago
  You can get pretty far without any of that fancy stuff. You can get plenty done by using parallel-ssh and then focusing on the actual thing you develop instead of endless tooling and docker and terraform and kubernetes and salt and puppet and ansible. Sure, if you know why you need them and know what value you get from them OK. But many people just do it because it's the thing to do...
- marcosdumay 2y ago
  All of that is... completely unrelated to the GP's post. Did you reply to the right comment? Do you think "politics" is something you solve with Ansible?
  - sgarland 2y ago
    > Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. It's related to the first part. Re: the second, IME if you let dev teams run wild with "managing their own infra," the org as a whole eventually pays for that when the dozen bespoke stacks all hit various bottlenecks, and no one actually understands how they work, or how to troubleshoot them. I keep being told that "reducing friction" and "increasing velocity" are good things; I vehemently disagree. It might be good for short-term profits, but it is poison for long-term success.
    - marcosdumay 2y ago
      > I keep being told that "reducing friction" and "increasing velocity" are good things As always, good rules are good, and bad rules are bad. Like most people on the internet, you are assuming only one of those sets exist. But you are just assuming a different set from everybody that you are criticizing.
- sanderjd 2y ago
  I have never ever worked somewhere with one of these "cloud-like but custom on our own infrastructure" setups that didn't leak infrastructure concerns through the abstraction, to a significantly larger degree than AWS. I believe it can work, so maybe there are really successful implementations of this out there, I just haven't seen it myself yet!
daemonologist 2y ago
Our big company locked all cloud resources behind a floating/company-wide DevOps team (git and CI too). We have an old on-prem server that we jealously guard because it allows us to create remotes for new git repos and deploy prototypes without consulting anyone. (To be fair, I can see why they did it - a lot of deployments were an absolute mess before.)
mark242 2y ago
This is absolutely spot on. What do you mean, I can't scale up because I've used my hardware capex budget for the year?

by zosima 2y ago

Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. This is worth astronomical amounts of money in big corps.

acedTrex 2y ago

I have said for years the value of cloud is mainly its api, thats the selling point in large enterprise.

sgarland 2y ago
Self-hosted software also has APIs, and Terraform libraries, and Ansible playbooks, etc. It’s just that you have to know what it is you’re trying to do, instead of asking AWS what collection of XaaS you should use.

sgarland 2y ago

I’m not convinced this is entirely true. The upfront cost if you don’t have the skills, sure – it takes time to learn Linux administration, not to mention management tooling like Ansible, Puppet, etc. But once those are set up, how is it different? AWS is quite clear with their responsibility model that you still have to tune your DB, for example. And for the setup, just as there are Terraform modules to do everything under the sun, there are Ansible (or Chef, or Salt…) playbooks to do the same. For both, you _should_ know what all of the options are doing. The only way I see this sentiment being true is that a dev team, with no infrastructure experience, can more easily spin up a lot of infra – likely in a sub-optimal fashion – to run their application. When it inevitably breaks, they can then throw money at the problem via vertical scaling, rather than addressing the root cause.

the__alchemist 2y ago
Do you need those tools? It seems that for fundamental web hosting, you need your application server, nginx or similar, postgres or similar, and a CLI. (And an interpreter etc if your application is in an interpreted lang)
- sgarland 2y ago
  I suppose that depends on your RTO. With cloud providers, even on a bare VM, you can to some extent get away with having no IaC, since your data (and therefore config) is almost certainly on networked storage which is redundant by design. If an EC2 fails, or even if one of the drives in your EBS drive fails, it'll probably come back up as it was. If it's your own hardware, if you don't have IaC of some kind – even something as crude as a shell script – then a failure may well mean you need to manually set everything up again.
  - noprocrasted 2y ago
    Get two servers (or three, etc)?
    - sgarland 2y ago
      Well, sure – I was trying to do a comparison in favor of cloud, because the fact that EBS Volumes can magically detach and attach is admittedly a neat trick. You can of course accomplish the same (to a certain scale) with distributed storage systems like Ceph, Longhorn, etc. but then you have to have multiple servers, and if you have multiple servers, you probably also have your application load balanced with failover.
  - champtar 2y ago
    All EBS volumes except io2 have advertised durability of 99.8%, which is pretty low, so don't count it in the magic networked storage category.
- zbentley 2y ago
  For fundamentals, that list is missing: - Some sort of firewall or network access control. Being able to say "allow http/s from the world (optionally minus some abuser IPs that cause problems), and allow SSH from developers (by IP, key, or both)" at a separate layer from nginx is prudent. Can be ip/tables config on servers or a separate firewall appliance. - Some mechanism of managing storage persistence for the database, e.g. backups, RAID, data files stored on fast network-attached storage, db-level replication. Not losing all user data if you lose the DB server is table stakes. - Something watching external logging or telemetry to let administrators know when errors (e.g. server failures, overload events, spikes in 500s returned) occur. This could be as simple as Pingdom or as involved as automated alerting based on load balancer metrics. Relying on users to report downtime events is not a good approach. - Some sort of CDN, for applications with a frontend component. This isn't required for fundamental web hosting, but for sites with a frontend and even moderate (10s/sec) hit rates, it can become required for cost/performance; CDNs help with egress congestion (and fees, if you're paying for metered bandwidth). - Some means of replacing infrastructure from nothing. If the server catches fire or the hosting provider nukes it, having a way to get back to where you were is important. Written procedures are fine if you can handle long downtime while replacing things, but even for a handful of application components those procedures get pretty lengthy, so you start wishing for automation. - Some mechanism for deploying new code, replacing infrastructure, or migrating data. Again, written procedures are OK, but start to become unwieldy very early on ('stop app, stop postgres, upgrade the postgres version, start postgres, then apply application migrations to ensure compatibility with new version of postgres, then start app--oops, forgot to take a postgres backup/forgot that upgrading postgres would break the replication stream, gotta write that down for net time...'). ...and that's just for a very, very basic web hosting application--one that doesn't need caches, blob stores, the ability to quickly scale out application server or database capacity. Each of those things can be accomplished the traditional way--and you're right, that sometimes that way is easier for a given item in the list (especially if your maintainers have expertise in that item)! But in aggregate, having a cloud provider handle each of those concerns tends to be easier overall and not require nearly as much in-house expertise.
zosima 2y ago
You are focusing on technology. And sure of course you can get most of the benefits of AWS a lot cheaper when self-hosting. But when you start factoring internal processes and incompetent IT departments, suddenly that's not actually a viable option in many real-world scenarios.
- jeffbee 2y ago
  Exactly. With the cloud you can suddenly do all the things your tyrannical Windows IT admin has been saying are impossible for the last 30 years.
  - the_arun 2y ago
    It is similar to cooking at home vs ordering cooked food everyday. If some guarantees the taste & quality people would happy to outsource it.
tylerchurch 2y ago
I think this is only true for teams and apps of a certain size. I've worked on plenty of teams with relatively small apps, and the difference between: 1. Cloud: "open up the cloud console and start a VM" 2. Owned hardware: "price out a server, order it, find a suitable datacenter, sign a contract, get it racked, etc." Is quite large. #1 is 15 minutes for a single team lead. #2 requires the team to agree on hardware specs, get management approval, finance approval, executives signing contracts. And through all this you don't have anything online yet for... weeks? If your team or your app is large, this probably all averages out in favor of #2. But small teams often don't have the bandwidth or the budget.
- noprocrasted 2y ago
  3. "Dedicated server" at any hosting provider Open their management console, press order now, 15 mins later get your server's IP address.
  - zbentley 2y ago
    For purposes of this discussion, isn't AWS just a very large hosting provider? I.e. most hosting providers give you the option for virtual or dedicated hardware. So does Amazon (metal instances). Like, "cloud" was always an ill-defined term, but in the case of "how do I provision full servers" I think there's no qualitative difference between Amazon and other hosting providers. Quantitative, sure.
    - noprocrasted 2y ago
      > Amazon (metal instances) But you still get nickel & dimed and pay insane costs, including on bandwidth (which is free in most conventional hosting providers, and overages are 90x cheaper than AWS' costs).
    - irunmyownemail 2y ago
      Qualitatively, AWS is greedy and nickle and dime you to death. Their Route53 service doesn't even have all the standard DNS options I need and I can get everywhere else or even on my own running bind9. I do not use IPv6 for several reasons, when AWS decided charge for IPv4, I went looking elsewhere to get my VM's. I can't even imagine how much the US Federal Government is charging American taxpayers to pay AWS for hosting there, it has to be astronomical.
bonoboTP 2y ago
You can get pretty far without any of that fancy stuff. You can get plenty done by using parallel-ssh and then focusing on the actual thing you develop instead of endless tooling and docker and terraform and kubernetes and salt and puppet and ansible. Sure, if you know why you need them and know what value you get from them OK. But many people just do it because it's the thing to do...
marcosdumay 2y ago
All of that is... completely unrelated to the GP's post. Did you reply to the right comment? Do you think "politics" is something you solve with Ansible?
- sgarland 2y ago
  > Cloud expands the capabilities of what one team can manage by themselves, enabling them to avoid a huge amount of internal politics. It's related to the first part. Re: the second, IME if you let dev teams run wild with "managing their own infra," the org as a whole eventually pays for that when the dozen bespoke stacks all hit various bottlenecks, and no one actually understands how they work, or how to troubleshoot them. I keep being told that "reducing friction" and "increasing velocity" are good things; I vehemently disagree. It might be good for short-term profits, but it is poison for long-term success.
  - marcosdumay 2y ago
    > I keep being told that "reducing friction" and "increasing velocity" are good things As always, good rules are good, and bad rules are bad. Like most people on the internet, you are assuming only one of those sets exist. But you are just assuming a different set from everybody that you are criticizing.
sanderjd 2y ago
I have never ever worked somewhere with one of these "cloud-like but custom on our own infrastructure" setups that didn't leak infrastructure concerns through the abstraction, to a significantly larger degree than AWS. I believe it can work, so maybe there are really successful implementations of this out there, I just haven't seen it myself yet!

daemonologist 2y ago

Our big company locked all cloud resources behind a floating/company-wide DevOps team (git and CI too). We have an old on-prem server that we jealously guard because it allows us to create remotes for new git repos and deploy prototypes without consulting anyone. (To be fair, I can see why they did it - a lot of deployments were an absolute mess before.)

mark242 2y ago

This is absolutely spot on. What do you mean, I can't scale up because I've used my hardware capex budget for the year?