7 ms·

Ah the old cloud provider switcheroo. Yip this is the way they make money. They make it easy to setup some gigantic hugely scalable website then hit you with a

Ah the old cloud provider switcheroo. Yip this is the way they make money. They make it easy to setup some gigantic hugely scalable website then hit you with a gigantic scaled up bill. AWS would do this as well. Team I'm in at the moment is in the early stages of cloud adoption but the company in total has fell hook line and sinker for AWS. When I mentioned the cost there is always an excuse. The main one being that you don't have to hire sysadmins anymore as that's taken care now by AWS. Ah yes but they have actually been replaced with a "DevOps" team plus just our department now spend > £1 million per year to AWS in hosting costs. A 20% reduction in those fees could pay for a few sysadmin(s). The next one is that no other vendor would be able to supply the kit. You know StackOverflow is able to run on a single webserver (https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...). Plus many of the other providers have loads of instances available. I mean I'm not against cloud it's just not the cheapest option if you choose one of the big 3 providers. I use a company called scaleway (https://www.scaleway.com/en/ https://www.scaleway.com/en/) they have all the essential cloud services you need and everything else you can run yourself in docker or k8s.

mcbain 5y ago
That stackoverflow infra blog post is out of date. They use more than a single webserver now. For example: https://stackexchange.com/performance https://stackexchange.com/performance
- andrewxdiamond 5y ago
  Most importantly, SO is extremely read-heavy, write-lite, and cache-friendly. A similar “scale” e-commerce site would be significantly more load, have more dynamic data, and just be overall harder to run.
- dijit 5y ago
  Now they have 9. They still serve a lot more traffic than I do and I have hundreds of instances; thousands of containers.
  - nightpool 5y ago
    You have thousands of containers? Physician, heal thyself.
    - folmar 5y ago
      Thousands is not a lot. If you do microservices and have 100 of them, 3 replicas of each for dev, qa and prod, you already are at 900.
    - dijit 5y ago
      I mean, at my last job I had thousands of physical machines too. Scale can depend on many things. Here's a couple of reasons why it can easily be thousands: 1) Cronjobs, CI jobs, ETL, FaaS are all systems that exist. What used to be a process is now a container. (one need only check the PID count on their local machine to know that this can be many quite easily). 2) Microservices; I'm a larger fan of fat "services" but doing actual micro services tends to leave you with a lot of containers running 3) Actual compute need. If my original hosting strategy was thousands of machines, well, I'm going to have thousands of containers, if not more.
      - nightpool 5y ago
        Sure, but the implied message of your comment that you were saying you could replace all of your instances and containers with just 9 machines, since StackOverflow "serves a lot more traffic than you do" (i.e. "has more actual compute need"). I think most reasonable engineers would say that "thousands" of containers would be a massive mistake to use for that size of task, even if few of them would go to the extent that Stack Overflow did of using only 9 machines.
- Dave3of5 5y ago
  Looks like they have actually reduces their footprint. It not that they do run on a single webserver it's that they can run on one.
  - chasd00 5y ago
    > Looks like they have actually reduces their footprint. i don't remember who said it but a quote i really like is "it's not finished when there's nothing left to add, it's finished when there's nothing left to take away"
    - mark-r 5y ago
      It's commonly attributed to Antoine de Saint-Exupéry and is a lot older than I thought, from 1935 and originally in French. https://english.stackexchange.com/q/38837/178351 https://english.stackexchange.com/q/38837/178351
3pt14159 5y ago
I can see both sides. If you're a startup that needs to be able to scale quickly if product market fit is achieved, the cloud really saves your bacon. Or is your ten person team really going to figure out how to get Postgres to reliably run with billions of records, with encrypted backups, etc? It's basically a form of permanent debt. Faster product market fit, higher long term infrastructure costs until you have enough breathing room to start pulling it into your own datacenter. At that point you have some negotiating leverage with the cloud provider. On the other hand, if you're not looking for explosive growth man oh man is DigitalOcean or anyone of a number good providers of good old VPSes / Cloud-lite.
- capableweb 5y ago
  I keep hearing this argument against using your own infrastructure again and again, and I'm not sure how true it is. I've worked with teams on both sides, and everyone is gonna have to deal with figuring out how to run at scale, it's just different ways of achieving that. I've worked with teams that manage their own infrastructure with dedicated servers, and not having to think about scaling for a long time as the one beefy server could just take whatever load you threw at it. I've also worked with teams who don't manage their own infrastructure and thought they were ready to scale without issues, but once the scale actually happened, it turned out there was more things to consider than just the amount of servers you run, race-conditions were everywhere but no one thought about that. Definitely a case of "right tool for the right job", but I don't think it's as easy as "Self-managed: harder to scale, PaaS/Cloud: easy peazy to scale".
  - Dave3of5 5y ago
    Yeah agreed I haven't worked with Google scale companies but I've always found scaling issue to to development related not infrastructure related. So examples would be a bad db query that takes the system down, overly chatting webserver that issues too many queries to the backend, pulling large datasets into the webapp causing exhaustion of memory ...etc. AWS / Azure can't be these issues they have to be fixed in your code. There is definitely a place for AWS/Azure and their offering of services is fantastic but they are not a silver bullet for scaling your website to millions of active user. On another point though the vast majority of websites you'll ever build won't have that level of active users. It's a good problem to have though as it means your site is doing really well.
traceroute66 5y ago
See also Let's Encrypt: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...
nova22033 5y ago
our department now spend > £1 million per year to AWS in hosting costs. A 20% reduction in those fees could pay for a few sysadmin(s). You can hire a "few" sysadmins for 200k/year?
- mattbee 5y ago
  £200k / year, in the UK? That's about 2-5 depending on experience.
- Dave3of5 5y ago
  In the UK/Europe yes: https://uk.indeed.com/jobs?q=System%20Administrator&vjk=5149fbd5d9cc0521 https://uk.indeed.com/jobs?q=System%20Administrator&vjk=5149... Probably not at FAANG level salaries but I doubt there are many sysadmins working for FAANG companies anymore. DevOps btw are more expensive and infact in the UK DevOps can be higher paid that a developer. I suspect most of the DevOps working for this company are on £65k+. According to: https://ifs.org.uk/tools_and_resources/where_do_you_fit_in https://ifs.org.uk/tools_and_resources/where_do_you_fit_in That puts those earners in the top 3% or from that website: " In the below graph, the alternatively shaded sections represent the different decile groups. As you can see, you are in the 10th decile group. In conclusion, Your income is so high that you lie beyond the far right hand side of the chart. "
- sparselogic 5y ago
  A 20% reduction would result in ~£800k/yr.
  - Arnavion 5y ago
    They're saying that if the AWS costs decreased by 20%, they could use the now freed-up money, ie 200k, to pay sysadmins.
Kneecaps07 5y ago
There's an argument to be made for quality of life for your employees. As someone who has transitioned from on-prem server management to mainly cloud work, my job happiness has skyrocketed. I haven't set foot in a data center in three years and I do not miss it one bit. Dealing with hardware failures, hardware vendors, confusing licensing, having to know SKUs, racking new cabinets, swapping hard drives, patching servers - it's all awful work. When you go cloud only, you can be more productive instead of dealing with some of that nonsense work.
- Dave3of5 5y ago
  I think this depends. For OPS people no longer having to physically go into a DC I agree but you've now pushed a bunch of work developers especially now will have a harder time as they used to make code and there was someone who sorted infrastructure now the devs themselves are kept up all nights with AWS stuff going up and down. If cloud improved QOL for ALL employees I'd agree but I think it just shifts work around and costs more.
- Symbiote 5y ago
  In between your two extremes are colocation (no managing buildings, power, cooling, racks, security, optionally network), dedicated servers (no managing/installing servers, disks, warranties) and basic VMs.
  - agsamek 5y ago
    We do colocation and we have to deal with HD and ram failures from time to time. Replacement of the hardware part is managed by the provider, but discovery and software requieres our involvment. I just wonder what happens if a ram or hd failure hits a cloud provider node. Is the architecture on average really able to come over such failures without help and intervention.
- BlueTemplar 5y ago
  It's not like all those jobs have been taken over by automation - someone still has to take care of these cloud servers ?
- 5y ago
InefficientRed 5y ago
> > £1 million per year I'm curious about your workload. I tend to only use cloud for workloads where it's either (1) by far the only feasible option (e.g. need GPUs for short periods of time), or else (2) basically free. > I mean I'm not against cloud it's just not the cheapest option This is certainly true for most workloads. It's also true that buying is better than renting, but here I am living in a rented apartment. The logic from on high might be something like "if demand is uncertain and capex is risky, why buy when you can rent?"

mcbain 5y ago

That stackoverflow infra blog post is out of date. They use more than a single webserver now. For example: https://stackexchange.com/performance https://stackexchange.com/performance

andrewxdiamond 5y ago
Most importantly, SO is extremely read-heavy, write-lite, and cache-friendly. A similar “scale” e-commerce site would be significantly more load, have more dynamic data, and just be overall harder to run.
dijit 5y ago
Now they have 9. They still serve a lot more traffic than I do and I have hundreds of instances; thousands of containers.
- nightpool 5y ago
  You have thousands of containers? Physician, heal thyself.
  - folmar 5y ago
    Thousands is not a lot. If you do microservices and have 100 of them, 3 replicas of each for dev, qa and prod, you already are at 900.
  - dijit 5y ago
    I mean, at my last job I had thousands of physical machines too. Scale can depend on many things. Here's a couple of reasons why it can easily be thousands: 1) Cronjobs, CI jobs, ETL, FaaS are all systems that exist. What used to be a process is now a container. (one need only check the PID count on their local machine to know that this can be many quite easily). 2) Microservices; I'm a larger fan of fat "services" but doing actual micro services tends to leave you with a lot of containers running 3) Actual compute need. If my original hosting strategy was thousands of machines, well, I'm going to have thousands of containers, if not more.
    - nightpool 5y ago
      Sure, but the implied message of your comment that you were saying you could replace all of your instances and containers with just 9 machines, since StackOverflow "serves a lot more traffic than you do" (i.e. "has more actual compute need"). I think most reasonable engineers would say that "thousands" of containers would be a massive mistake to use for that size of task, even if few of them would go to the extent that Stack Overflow did of using only 9 machines.
Dave3of5 5y ago
Looks like they have actually reduces their footprint. It not that they do run on a single webserver it's that they can run on one.
- chasd00 5y ago
  > Looks like they have actually reduces their footprint. i don't remember who said it but a quote i really like is "it's not finished when there's nothing left to add, it's finished when there's nothing left to take away"
  - mark-r 5y ago
    It's commonly attributed to Antoine de Saint-Exupéry and is a lot older than I thought, from 1935 and originally in French. https://english.stackexchange.com/q/38837/178351 https://english.stackexchange.com/q/38837/178351

3pt14159 5y ago

I can see both sides. If you're a startup that needs to be able to scale quickly if product market fit is achieved, the cloud really saves your bacon. Or is your ten person team really going to figure out how to get Postgres to reliably run with billions of records, with encrypted backups, etc? It's basically a form of permanent debt. Faster product market fit, higher long term infrastructure costs until you have enough breathing room to start pulling it into your own datacenter. At that point you have some negotiating leverage with the cloud provider. On the other hand, if you're not looking for explosive growth man oh man is DigitalOcean or anyone of a number good providers of good old VPSes / Cloud-lite.

capableweb 5y ago
I keep hearing this argument against using your own infrastructure again and again, and I'm not sure how true it is. I've worked with teams on both sides, and everyone is gonna have to deal with figuring out how to run at scale, it's just different ways of achieving that. I've worked with teams that manage their own infrastructure with dedicated servers, and not having to think about scaling for a long time as the one beefy server could just take whatever load you threw at it. I've also worked with teams who don't manage their own infrastructure and thought they were ready to scale without issues, but once the scale actually happened, it turned out there was more things to consider than just the amount of servers you run, race-conditions were everywhere but no one thought about that. Definitely a case of "right tool for the right job", but I don't think it's as easy as "Self-managed: harder to scale, PaaS/Cloud: easy peazy to scale".
- Dave3of5 5y ago
  Yeah agreed I haven't worked with Google scale companies but I've always found scaling issue to to development related not infrastructure related. So examples would be a bad db query that takes the system down, overly chatting webserver that issues too many queries to the backend, pulling large datasets into the webapp causing exhaustion of memory ...etc. AWS / Azure can't be these issues they have to be fixed in your code. There is definitely a place for AWS/Azure and their offering of services is fantastic but they are not a silver bullet for scaling your website to millions of active user. On another point though the vast majority of websites you'll ever build won't have that level of active users. It's a good problem to have though as it means your site is doing really well.

traceroute66 5y ago

See also Let's Encrypt: https://letsencrypt.org/2021/01/21/next-gen-database-servers.html https://letsencrypt.org/2021/01/21/next-gen-database-servers...

nova22033 5y ago

our department now spend > £1 million per year to AWS in hosting costs. A 20% reduction in those fees could pay for a few sysadmin(s). You can hire a "few" sysadmins for 200k/year?

mattbee 5y ago
£200k / year, in the UK? That's about 2-5 depending on experience.
Dave3of5 5y ago
In the UK/Europe yes: https://uk.indeed.com/jobs?q=System%20Administrator&vjk=5149fbd5d9cc0521 https://uk.indeed.com/jobs?q=System%20Administrator&vjk=5149... Probably not at FAANG level salaries but I doubt there are many sysadmins working for FAANG companies anymore. DevOps btw are more expensive and infact in the UK DevOps can be higher paid that a developer. I suspect most of the DevOps working for this company are on £65k+. According to: https://ifs.org.uk/tools_and_resources/where_do_you_fit_in https://ifs.org.uk/tools_and_resources/where_do_you_fit_in That puts those earners in the top 3% or from that website: " In the below graph, the alternatively shaded sections represent the different decile groups. As you can see, you are in the 10th decile group. In conclusion, Your income is so high that you lie beyond the far right hand side of the chart. "
sparselogic 5y ago
A 20% reduction would result in ~£800k/yr.
- Arnavion 5y ago
  They're saying that if the AWS costs decreased by 20%, they could use the now freed-up money, ie 200k, to pay sysadmins.

Kneecaps07 5y ago

There's an argument to be made for quality of life for your employees. As someone who has transitioned from on-prem server management to mainly cloud work, my job happiness has skyrocketed. I haven't set foot in a data center in three years and I do not miss it one bit. Dealing with hardware failures, hardware vendors, confusing licensing, having to know SKUs, racking new cabinets, swapping hard drives, patching servers - it's all awful work. When you go cloud only, you can be more productive instead of dealing with some of that nonsense work.

Dave3of5 5y ago
I think this depends. For OPS people no longer having to physically go into a DC I agree but you've now pushed a bunch of work developers especially now will have a harder time as they used to make code and there was someone who sorted infrastructure now the devs themselves are kept up all nights with AWS stuff going up and down. If cloud improved QOL for ALL employees I'd agree but I think it just shifts work around and costs more.
Symbiote 5y ago
In between your two extremes are colocation (no managing buildings, power, cooling, racks, security, optionally network), dedicated servers (no managing/installing servers, disks, warranties) and basic VMs.
- agsamek 5y ago
  We do colocation and we have to deal with HD and ram failures from time to time. Replacement of the hardware part is managed by the provider, but discovery and software requieres our involvment. I just wonder what happens if a ram or hd failure hits a cloud provider node. Is the architecture on average really able to come over such failures without help and intervention.
BlueTemplar 5y ago
It's not like all those jobs have been taken over by automation - someone still has to take care of these cloud servers ?
5y ago

InefficientRed 5y ago

> > £1 million per year I'm curious about your workload. I tend to only use cloud for workloads where it's either (1) by far the only feasible option (e.g. need GPUs for short periods of time), or else (2) basically free. > I mean I'm not against cloud it's just not the cheapest option This is certainly true for most workloads. It's also true that buying is better than renting, but here I am living in a rented apartment. The logic from on high might be something like "if demand is uncertain and capex is risky, why buy when you can rent?"