7 ms·

> I work for a <1000 employee firm, and AFAIK we're spending in 6 figures monthly with AWS. The argument for 6 figure cloud spend was always "it'll help you no

> I work for a <1000 employee firm, and AFAIK we're spending in 6 figures monthly with AWS. The argument for 6 figure cloud spend was always "it'll help you not need as many infrastructure-related dev ops engineers", right? Would you say that's true for your org? Every org I've ever worked at has had 6 figure monthly cloud bill, and the 6 figure monthly payroll bill related to teams working with cloud infrastructure

dragonwriter 4y ago
A six figure payroll bill related to teams working with cloud infrastructure? So, a low-single-digit number of engineers?
bastardoperator 4y ago
Considering the cost of the infra it would take to replicate some of these services, not to mention bandwidth, DC, and electricity costs, you're probably still saving money. You're absolutely saving on Network Ops and Data Center Ops not to mention the huge investment in gear one has to make. A single server can cost over 100K. Network gear can costs way more. The fact that you don't have to make those investments is the allure. At a certain point, you might outgrow someones else's cloud and build your own, but that divide is a fairly large threshold to cross.
- scarmig 4y ago
  And as far as problems go, "we're so successful that it makes financial sense to build our own on-prem cloud" is a pretty good problem to have.
  - xmprt 4y ago
    I always figured it was the other way around. When you're small it's pretty easy to get by with a stupidly simple solution but as you grow you end up needing to spend much more to build something scalable and at that point, using the cloud makes sense. The biggest success that cloud providers have had is convincing users that they need to spend $100k and that a much simpler $5k solution that's built using off the shelf components just won't cut it.
    - foobarian 4y ago
      The most painful is having to run multiple data centers for HA. Double or triple the price right there.
      - traceroute66 4y ago
        > The most painful is having to run multiple data centers for HA. Double or triple the price right there. Ok, let's make one thing patently clear: ITS THE SAME IN THE CLOUD All the cloud vendors will tell you need to have stuff replicated in multiple "Availability Zones" or "Regions". And yup, the nickle & diming nature of the cloud means that's going to cost you double or triple.
        foobarian 4y ago
        It's not though. With your own stuff you have at least one DC sitting idle, with all that private gear doing nothing. Doesn't matter if you don't use a single byte of bandwidth. With AWS at least some of that is not there.
        Dylan16807 4y ago
        If you're set up for HA you're paying for the idle hardware either way, and if you save on electricity that might benefit the DC option but not the cloud option. Overall not much difference there. Bandwidth is the one thing where the cloud clearly wins wrt idle servers... except that DC bandwidth is a hundred times cheaper than AWS bandwidth, so you should prefer buying 133% or 150% or 200% DC bandwidth by a mile.
        scarface74 4y ago
        Whether you are paying for HA depends on your Recovery Time Objective (RTO). You can have a bunch of suspended EC2 instances and non EC2 resources where you only pay per use in another region. You can redirect traffic to another region and have autoscaling spin up EC2 instances, etc.
- phpisthebest 4y ago
  >>A single server can cost over 100K. Network gear can costs way more. "Can" sure... likely to for most orgs no. Certainly not for someone running a 5 figure cloud bill... as a reference, our standard compute server we use on prem is $12,000 including a 5 year support contract. > At a certain point, you might outgrow someones else's cloud and build your own, If your buying $100K servers, and $100K network devices, I pretty sure you are at the point
  - baq 4y ago
    if you're buying either, you're paying millions in salaries to technicians and sysadmins and DBAs^W^W^Wdevops and SREs. i've been on both sides of the fence and it's a case of 'grass is always greener on the other side'. the truth is, running any sort of non-trivial infra is ducking expensive.
    - bcrosby95 4y ago
      This is what always made me chuckle a bit. I have a half dozen friends who were sysadmins 20 years ago and today they're "DevOps". Precisely zero have been put out of work.
      - throwawayapples 4y ago
        Yep. It's not like either sysadmin or devops are harder than or easier than the other... they're just different. Actually, seems like managing k8s is an order of time expenditure greater than managing an old-school F5 with a bunch of Unixy web servers behind it.
        primax 4y ago
        When I was a sysadmin it was definitely harder. Devops gives the benefit of cows instead of pets, and a ton of reusable work. So I get why it's more 'valuable' from a remuneration basis.
        4y ago
- jbverschoor 4y ago
  A single server which can probably serve your complete customer base could / might cost less than 5k
  - nathancahill 4y ago
    As I recall, StackOverflow runs (ran?) on 6 large but not massive servers.
    - ctvo 4y ago
      They also used the dotnet stack (Windows Server, IIS, MSSQL, dotnet), and optimized the heck out of everything. They're not the typical use case. (I'm not saying dotnet allowed them to get by on N low digit servers. I'm saying those folks are atypical)
      - l33t233372 4y ago
        What are you saying by pointing out that they use the dotnet stack? It’s interesting trivia to be sure, but I’m wondering if you were making a point with that
        drchickensalad 4y ago
        Have you seen the performance of standard ruby and python web frameworks in comparison? It's a massive difference
      - bcrosby95 4y ago
        We did very similar with a Java stack without even trying really. Competitors using things like Ruby and went all in on distributed messes had hundreds of servers but we had about 15. It does require you to be aware of performance, but I wouldn't call it difficult or particularly time consuming.
      - ilyt 4y ago
        So assuming your code is 5 times worse that still fits within one rack > I'm saying those folks are atypical) Calling "write not shit code in language that's not dog slow" "atypical" is sad state of our industry. Also in many cases you can get a lot out of caching if you do it smartly.
- dimitrios1 4y ago
  Having worked half my career at places with their own data centers and self ran infra, and the other half with mostly cloud based solutions, I have a theory. Perhaps we are designing far more complicated solutions now to leverage these cloud services, whereas having the constraints of a self operated data center and infrastructure necessitates more ingenuity to achieve similar results. We used to do so much more with just a few pieces of infrastructure, like our RDBMS's, as one example. It was amazing to me how many scenarios we solved with just a couple of solid vertically scaled database servers with active-active failovers, Redis, an on-prem load balancer, and some webservers (later, self hosted containerization software). We used to design for as few infrastructure pieces as possible, now it seems like that is rarely a constraint people have in their minds anymore.
  - notyourwork 4y ago
    Amen, I'm becoming an old grumpy engineer on my team for constantly asking why we need yet another <insert cloud technology here>. I'm not against new technology but I am against not considering what we have and how it may already solve the problem without adding wider breadth to our operational surface area. And it's every single damn year now because now cloud providers string their own cloud primitives together to form new cloud services.
    - datavirtue 4y ago
      Could have just called an API but instead we fired an SNS event. Sigh.
      - notyourwork 4y ago
        How many times I've had this discussion. Let's publish a notification, and let's have the message receiver call some API. Why not just call the API from the place where you want to publish the message? Because we need this SNS message queue.
        jquast 4y ago
- ilyt 4y ago
  > A single server can cost over 100K. Network gear can costs way more. The fact that you don't have to make those investments is the allure. IF you're pushing 400+ Gbit, sure. Most won't.
renewiltord 4y ago
The real cost in most places is in opportunity cost and lead time. e.g. you want to do a thing but you can't because you don't have the physical resources available. That can be very frustrating for people trying out things that will produce novel revenue in the org. Reducing the cost of experimentation is quite valuable in a larger organization. For something like our HFT firm, we see high sustained utilization so we have an on-prem cluster of nodes that we run. They're faster (more powerful CPUs), have more RAM, and you can run fiber optic interconnect that's, in practice, faster than on AWS. We still use AWS extensively, though.
- nostrebored 4y ago
  You can use fiber for hft through AWS as well, but you lose a lot of the benefits of cloud when you have a sustained workload that doesn’t benefit from being fault tolerant because the system would be down anyway.
- candiodari 4y ago
  Except that for the same cost in cloud you can't just have double the resources in dedicated servers, but like 3x to 10x depending. And if you do colo, even more. I get that if you have management that won't keep even 10% spare capacity online you can't work like this, but they will also "tightly control cloud spending", won't they? Not sure. Cloud only really makes sense at really small scale.
  - renewiltord 4y ago
    Everyone's use cases are different. But I think I'm making the right trade-offs here overall (I am management in this world).
    - deleted 4y ago
      [deleted]
crabbone 4y ago
It's kind of even worse: since public cloud became very popular, the profile of infrastructure programmers changed. A lot of knowledge necessary to run own infrastructure disappeared, being replaced by the knowledge of how to operate the service. So, for example, things like PXE boot, which would be useful in your own datacenter, but don't work (well) in clouds s.a. AWS become mystery. Obviously, this isn't limited to just PXE. And since public cloud's goal is to provide service to as many customers as possible while having as few staff infra engineers, the total expertise among general public disappears. Having worked in the infra business for a while -- it's increasingly more difficult to hire infra engineers.
- Kye 4y ago
  Is there some institutional barrier to training new hires? It seems like the inevitable result if you need an increasingly niche skill. The situation doesn't seem like it'll get better on its own.
  - bcrosby95 4y ago
    Train by who? How do you start this process? If you never had the expertise, or more unfortunately had it but lost it, it's very expensive to gain it.
    - Kye 4y ago
      This seems to be a growing problem across the economy. Companies weren't willing to train for so long that now not only is there no one to hire who knows how to do jobs, there's no one left to train people to do them.
  - crabbone 4y ago
    I work in an infra department, which used to be its own company before it was bought by a bigger one. Out of about 20 developers, one is younger than 30. Most are in their 40s and some are 50+. I know that when I was hired, my job was advertised for over a year, if not longer. They pay may not be stellar, but it's still pretty decent... In my previous position, I was also responsible for screening and hiring, so I also had some insight into what's available on the job market. My impression is that the younger generation simply doesn't even know this kind of job exists. Those who go into programming aspire to be a Web developer, or an ML developer, or a game developer... DevOps is one of those things people have heard about. But, this label was coopted by cloud some time ago. So, those aiming for DevOps role think they need to learn how to use public cloud API... It's actually quite eye-opening to have access to the hiring process. One of the very surprising things to me was that newer generation of programmers tends to be confused about persistent / volatile storage. So, if you asked them to give a description of what a "file" is, more than once I've heard an answer like "a portion of memory". Which, while true in some sense, is a weird way to put it. But then I realized that people who are entering the field today may not have been exposed to PCs. Naively, if anyone asked me to show where the files in the computer live, I'd get them a picture of a hard drive. But people who only ever used laptops probably don't even know what a hard drive looks like. Finally, it's also a chicken and egg problem. You cannot really learn about infra outside of a big org that has its own physical infra. Similar to how you'd never be exposed to configuring BGP if all your exposure to computers is your own PC. A popular method of learning today is to learn online. But, if you do that -- you are almost inevitably driven to the public cloud, because that's the way to provide you with all that virtual infrastructure online. You cannot also practice at home, since the hardware to practice on tends to be very expensive and not very useful for individual users. Other fields get it easier because they have to select from somewhat prepared candidates. When it comes to infra, it's a lot more investment into teaching one.
lightbendover 4y ago
> Every org I've ever worked at has had 6 figure monthly cloud bill, and the 6 figure monthly payroll bill related to teams working with cloud infrastructure There was a point not too long ago when I managed a 6 engineer team responsible for 7 figures of monthly spend. Still seems outlandish to me despite the ops burden being pretty low once we had things sorted.

dragonwriter 4y ago

A six figure payroll bill related to teams working with cloud infrastructure? So, a low-single-digit number of engineers?

bastardoperator 4y ago

Considering the cost of the infra it would take to replicate some of these services, not to mention bandwidth, DC, and electricity costs, you're probably still saving money. You're absolutely saving on Network Ops and Data Center Ops not to mention the huge investment in gear one has to make. A single server can cost over 100K. Network gear can costs way more. The fact that you don't have to make those investments is the allure. At a certain point, you might outgrow someones else's cloud and build your own, but that divide is a fairly large threshold to cross.

scarmig 4y ago
And as far as problems go, "we're so successful that it makes financial sense to build our own on-prem cloud" is a pretty good problem to have.
- xmprt 4y ago
  I always figured it was the other way around. When you're small it's pretty easy to get by with a stupidly simple solution but as you grow you end up needing to spend much more to build something scalable and at that point, using the cloud makes sense. The biggest success that cloud providers have had is convincing users that they need to spend $100k and that a much simpler $5k solution that's built using off the shelf components just won't cut it.
  - foobarian 4y ago
    The most painful is having to run multiple data centers for HA. Double or triple the price right there.
    - traceroute66 4y ago
      > The most painful is having to run multiple data centers for HA. Double or triple the price right there. Ok, let's make one thing patently clear: ITS THE SAME IN THE CLOUD All the cloud vendors will tell you need to have stuff replicated in multiple "Availability Zones" or "Regions". And yup, the nickle & diming nature of the cloud means that's going to cost you double or triple.
      - foobarian 4y ago
        It's not though. With your own stuff you have at least one DC sitting idle, with all that private gear doing nothing. Doesn't matter if you don't use a single byte of bandwidth. With AWS at least some of that is not there.
        Dylan16807 4y ago
        If you're set up for HA you're paying for the idle hardware either way, and if you save on electricity that might benefit the DC option but not the cloud option. Overall not much difference there. Bandwidth is the one thing where the cloud clearly wins wrt idle servers... except that DC bandwidth is a hundred times cheaper than AWS bandwidth, so you should prefer buying 133% or 150% or 200% DC bandwidth by a mile.
        scarface74 4y ago
        Whether you are paying for HA depends on your Recovery Time Objective (RTO). You can have a bunch of suspended EC2 instances and non EC2 resources where you only pay per use in another region. You can redirect traffic to another region and have autoscaling spin up EC2 instances, etc.
phpisthebest 4y ago
>>A single server can cost over 100K. Network gear can costs way more. "Can" sure... likely to for most orgs no. Certainly not for someone running a 5 figure cloud bill... as a reference, our standard compute server we use on prem is $12,000 including a 5 year support contract. > At a certain point, you might outgrow someones else's cloud and build your own, If your buying $100K servers, and $100K network devices, I pretty sure you are at the point
- baq 4y ago
  if you're buying either, you're paying millions in salaries to technicians and sysadmins and DBAs^W^W^Wdevops and SREs. i've been on both sides of the fence and it's a case of 'grass is always greener on the other side'. the truth is, running any sort of non-trivial infra is ducking expensive.
  - bcrosby95 4y ago
    This is what always made me chuckle a bit. I have a half dozen friends who were sysadmins 20 years ago and today they're "DevOps". Precisely zero have been put out of work.
    - throwawayapples 4y ago
      Yep. It's not like either sysadmin or devops are harder than or easier than the other... they're just different. Actually, seems like managing k8s is an order of time expenditure greater than managing an old-school F5 with a bunch of Unixy web servers behind it.
      - primax 4y ago
        When I was a sysadmin it was definitely harder. Devops gives the benefit of cows instead of pets, and a ton of reusable work. So I get why it's more 'valuable' from a remuneration basis.
        4y ago
jbverschoor 4y ago
A single server which can probably serve your complete customer base could / might cost less than 5k
- nathancahill 4y ago
  As I recall, StackOverflow runs (ran?) on 6 large but not massive servers.
  - ctvo 4y ago
    They also used the dotnet stack (Windows Server, IIS, MSSQL, dotnet), and optimized the heck out of everything. They're not the typical use case. (I'm not saying dotnet allowed them to get by on N low digit servers. I'm saying those folks are atypical)
    - l33t233372 4y ago
      What are you saying by pointing out that they use the dotnet stack? It’s interesting trivia to be sure, but I’m wondering if you were making a point with that
      - drchickensalad 4y ago
        Have you seen the performance of standard ruby and python web frameworks in comparison? It's a massive difference
    - bcrosby95 4y ago
      We did very similar with a Java stack without even trying really. Competitors using things like Ruby and went all in on distributed messes had hundreds of servers but we had about 15. It does require you to be aware of performance, but I wouldn't call it difficult or particularly time consuming.
    - ilyt 4y ago
      So assuming your code is 5 times worse that still fits within one rack > I'm saying those folks are atypical) Calling "write not shit code in language that's not dog slow" "atypical" is sad state of our industry. Also in many cases you can get a lot out of caching if you do it smartly.
dimitrios1 4y ago
Having worked half my career at places with their own data centers and self ran infra, and the other half with mostly cloud based solutions, I have a theory. Perhaps we are designing far more complicated solutions now to leverage these cloud services, whereas having the constraints of a self operated data center and infrastructure necessitates more ingenuity to achieve similar results. We used to do so much more with just a few pieces of infrastructure, like our RDBMS's, as one example. It was amazing to me how many scenarios we solved with just a couple of solid vertically scaled database servers with active-active failovers, Redis, an on-prem load balancer, and some webservers (later, self hosted containerization software). We used to design for as few infrastructure pieces as possible, now it seems like that is rarely a constraint people have in their minds anymore.
- notyourwork 4y ago
  Amen, I'm becoming an old grumpy engineer on my team for constantly asking why we need yet another <insert cloud technology here>. I'm not against new technology but I am against not considering what we have and how it may already solve the problem without adding wider breadth to our operational surface area. And it's every single damn year now because now cloud providers string their own cloud primitives together to form new cloud services.
  - datavirtue 4y ago
    Could have just called an API but instead we fired an SNS event. Sigh.
    - notyourwork 4y ago
      How many times I've had this discussion. Let's publish a notification, and let's have the message receiver call some API. Why not just call the API from the place where you want to publish the message? Because we need this SNS message queue.
      - jquast 4y ago
ilyt 4y ago
> A single server can cost over 100K. Network gear can costs way more. The fact that you don't have to make those investments is the allure. IF you're pushing 400+ Gbit, sure. Most won't.

renewiltord 4y ago

The real cost in most places is in opportunity cost and lead time. e.g. you want to do a thing but you can't because you don't have the physical resources available. That can be very frustrating for people trying out things that will produce novel revenue in the org. Reducing the cost of experimentation is quite valuable in a larger organization. For something like our HFT firm, we see high sustained utilization so we have an on-prem cluster of nodes that we run. They're faster (more powerful CPUs), have more RAM, and you can run fiber optic interconnect that's, in practice, faster than on AWS. We still use AWS extensively, though.

nostrebored 4y ago
You can use fiber for hft through AWS as well, but you lose a lot of the benefits of cloud when you have a sustained workload that doesn’t benefit from being fault tolerant because the system would be down anyway.
candiodari 4y ago
Except that for the same cost in cloud you can't just have double the resources in dedicated servers, but like 3x to 10x depending. And if you do colo, even more. I get that if you have management that won't keep even 10% spare capacity online you can't work like this, but they will also "tightly control cloud spending", won't they? Not sure. Cloud only really makes sense at really small scale.
- renewiltord 4y ago
  Everyone's use cases are different. But I think I'm making the right trade-offs here overall (I am management in this world).
  - deleted 4y ago
    [deleted]

crabbone 4y ago

It's kind of even worse: since public cloud became very popular, the profile of infrastructure programmers changed. A lot of knowledge necessary to run own infrastructure disappeared, being replaced by the knowledge of how to operate the service. So, for example, things like PXE boot, which would be useful in your own datacenter, but don't work (well) in clouds s.a. AWS become mystery. Obviously, this isn't limited to just PXE. And since public cloud's goal is to provide service to as many customers as possible while having as few staff infra engineers, the total expertise among general public disappears. Having worked in the infra business for a while -- it's increasingly more difficult to hire infra engineers.

Kye 4y ago
Is there some institutional barrier to training new hires? It seems like the inevitable result if you need an increasingly niche skill. The situation doesn't seem like it'll get better on its own.
- bcrosby95 4y ago
  Train by who? How do you start this process? If you never had the expertise, or more unfortunately had it but lost it, it's very expensive to gain it.
  - Kye 4y ago
    This seems to be a growing problem across the economy. Companies weren't willing to train for so long that now not only is there no one to hire who knows how to do jobs, there's no one left to train people to do them.
- crabbone 4y ago
  I work in an infra department, which used to be its own company before it was bought by a bigger one. Out of about 20 developers, one is younger than 30. Most are in their 40s and some are 50+. I know that when I was hired, my job was advertised for over a year, if not longer. They pay may not be stellar, but it's still pretty decent... In my previous position, I was also responsible for screening and hiring, so I also had some insight into what's available on the job market. My impression is that the younger generation simply doesn't even know this kind of job exists. Those who go into programming aspire to be a Web developer, or an ML developer, or a game developer... DevOps is one of those things people have heard about. But, this label was coopted by cloud some time ago. So, those aiming for DevOps role think they need to learn how to use public cloud API... It's actually quite eye-opening to have access to the hiring process. One of the very surprising things to me was that newer generation of programmers tends to be confused about persistent / volatile storage. So, if you asked them to give a description of what a "file" is, more than once I've heard an answer like "a portion of memory". Which, while true in some sense, is a weird way to put it. But then I realized that people who are entering the field today may not have been exposed to PCs. Naively, if anyone asked me to show where the files in the computer live, I'd get them a picture of a hard drive. But people who only ever used laptops probably don't even know what a hard drive looks like. Finally, it's also a chicken and egg problem. You cannot really learn about infra outside of a big org that has its own physical infra. Similar to how you'd never be exposed to configuring BGP if all your exposure to computers is your own PC. A popular method of learning today is to learn online. But, if you do that -- you are almost inevitably driven to the public cloud, because that's the way to provide you with all that virtual infrastructure online. You cannot also practice at home, since the hardware to practice on tends to be very expensive and not very useful for individual users. Other fields get it easier because they have to select from somewhat prepared candidates. When it comes to infra, it's a lot more investment into teaching one.

lightbendover 4y ago

> Every org I've ever worked at has had 6 figure monthly cloud bill, and the 6 figure monthly payroll bill related to teams working with cloud infrastructure There was a point not too long ago when I managed a 6 engineer team responsible for 7 figures of monthly spend. Still seems outlandish to me despite the ops burden being pretty low once we had things sorted.