7 ms·

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If t

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they have to hire ten additional ops engineers to run it at a cost of $1 million per year, then they've actually gone and burned $200k on something that provides no additional value to the business or their customers.

Dma54rhs 4y ago
AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.
- boltzmann-brain 4y ago
  Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.
  - monkpit 4y ago
    Brazil?
    - maxfurman 4y ago
      This is a reference to the film Brazil[0], which centers on a labyrinthine bureaucracy [0] https://www.imdb.com/title/tt0088846/ https://www.imdb.com/title/tt0088846/
      - dalmo3 4y ago
        > labyrinthine bureaucracy This is a reference to the country Brazil [0], which centers on... [0] https://www.bbc.com/news/business-18020623 https://www.bbc.com/news/business-18020623
        boltzmann-brain 4y ago
        > labyrinthine bureaucracy This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk [0] https://www.workhuman.com/blog/a-brief-history-of-bureaucracy-with-gary-hamel/ https://www.workhuman.com/blog/a-brief-history-of-bureaucrac... [1] https://en.wikipedia.org/wiki/Labyrinth https://en.wikipedia.org/wiki/Labyrinth
      - CreepGin 4y ago
        Wow! I thought it was a typo for "bizarre"... but now I know
      - earleybird 4y ago
        . . . and ducting https://www.youtube.com/watch?v=K9gO01pyv24 https://www.youtube.com/watch?v=K9gO01pyv24
    - malfist 4y ago
      Brazil is Amazon's internal build system. Not sure what GP is talking about here.
      - icedchai 4y ago
        You won't get the reference unless you've seen the movie.
  - deleted 4y ago
    [deleted]
  - moduspol 4y ago
    If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
    - saiya-jin 4y ago
      Not necessarily. Probably they already have pool of admins/devops that ran their systems for past 30 years, but have 0 AWS experience. Also, most companies don't need to scale infinitely, not everybody is building next google (in fact, almost nobody outside SV is).
    - Kon-Peki 4y ago
      You can buy on-prem managed storage solutions. Like PureStorage type things, where they call you up if something goes wrong, and are FedEx’ing you replacement parts before you even noticed that it had imperceptibly failed over to the standby power supply or whatever.
    - dijit 4y ago
      There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
      - icedchai 4y ago
        One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
  - neeleshs 4y ago
    The value prop of AWS/GCP etc go beyond bare metal vs VMs. Running HA databases (even if you are a startup, you need this), centralized logging, secrets manager, KMS, making sure disks are encrypted, something like pub/sub as a messaging backbone for your application, load balancers... the list goes on. Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
    - MajimasEyepatch 4y ago
      Not to mention compliance. If you have to go through third-party audits (and most businesses do after reaching a certain size), then using standard services from a public cloud provider can simplify things significantly. Sometimes that's because you can just offload "security of the cloud" to the cloud provider, and other times it's because there are well-established guidelines for how to achieve, say, PCI compliance on AWS.
    - Draiken 4y ago
      Yet most of these aren't needed for small crews... do I care that my DB is not HA when I have 1 RPS? You said yes, but I disagree. Do I need centralized logging if I have a few servers? It's already centralized for free. It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services. People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest. You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
      - neeleshs 4y ago
        YMMV, I run a startup that serves critical customer use cases, and from day 1 I had to care that my DB was HA, backed up, and could be restored. Actually had a case early on where we had to test this because I accidentally deleted customer data and had to restore - took all of one click, with no scripts, no prior investment into backup management etc. We also had to go through infosec reviews from day 1, and many of these are a must have. Doesn't matter if it's a small crew or large. Our customers care. 3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors. Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
otabdeveloper4 4y ago
Hiring a person to do 'docker compose up' for you is orders of magnitude cheaper than whatever AWS-specific knowldege is needed to not have AWS crap its bed.
- boltzmann-brain 4y ago
  not just "not have AWS crap its bed", but whatever sacrifices you need to perform in order for amazon not to come back with a bill that just so happens to be $1M higher than you expected it to be because of some gotcha that amazon profits off of, like (say) routing stupid traffic to one of their servers, currently occupied by you, that would be idle otherwise. Nice "mistake" to make.
- billythemaniam 4y ago
  If their AWS spend is $1M/year, it is not as simple as "docker compose up" on bare metal.
  - intelVISA 4y ago
    Might need some systemd as well then to be fair
tester756 4y ago
huge salaries + ten "ops engineers", lol. I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu
- citizenpaul 4y ago
  One time I was doing research on some really cheap data centers in basically 3rd world countries. Just out of curiosity to see how cheap it could really get. One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture? I thought it was strange they would put that picture on their site and not some GCI or something. I guess at least you know they are not lying about being a real place?
  - tester756 4y ago
    I mean, the one I'm talking about is relatively reasonable on infra they've decent building, data center tier, so power/network redundancy, etc, etc they're just cheap as fuck when it comes to people
    - MajimasEyepatch 4y ago
      People are the most important part of any operation! That's the last thing you want to cheap out on.
  - choletentent 4y ago
    I’ve just had a good laugh from your comment! Maybe water on the floor was for cooling.
  - loloquwowndueo 4y ago
    Share the company name / picture please :)
mbesto 4y ago
Exactly. In the "old IT world" we call this TCO = Total Cost of Ownership.
jjav 4y ago
> have to hire But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away. There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people. At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.
- antod 4y ago
  Yup, I worked for a company that moved everything from colo to AWS. Not only did the annual tech costs increase about 10x (even with all the AWS migration credit funny math), but they ended up with double the number of engineers to look after it. Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs. The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.
- doctor_eval 4y ago
  Agreed. I was doing some unrelated research yesterday and stumbled upon the fact that apparently (in Australia) 41% of companies see an increase in IT staff after adopting cloud services. [1] I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves. Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs. [1] https://www2.deloitte.com/au/en/pages/economics/articles/economic-value-cloud-services-australia.html https://www2.deloitte.com/au/en/pages/economics/articles/eco...
  - smugma 4y ago
jiveturkey 4y ago
You're not including reliability and availability projections, and the intangible cost of transferable skills wrt infrastructure. (ability to hire sufficiently skilled people to run it)
rstuart4133 4y ago
> If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, I guess you've never done it yourself? I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox. Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live. But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for. 10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.

Dma54rhs 4y ago

AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.

boltzmann-brain 4y ago
Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.
- monkpit 4y ago
  Brazil?
  - maxfurman 4y ago
    This is a reference to the film Brazil[0], which centers on a labyrinthine bureaucracy [0] https://www.imdb.com/title/tt0088846/ https://www.imdb.com/title/tt0088846/
    - dalmo3 4y ago
      > labyrinthine bureaucracy This is a reference to the country Brazil [0], which centers on... [0] https://www.bbc.com/news/business-18020623 https://www.bbc.com/news/business-18020623
      - boltzmann-brain 4y ago
        > labyrinthine bureaucracy This is a reference to "the rule of the desk" [0], and "an elaborate, confusing structure", which centers on a monster [1], in this case a bureaucrat at a desk [0] https://www.workhuman.com/blog/a-brief-history-of-bureaucracy-with-gary-hamel/ https://www.workhuman.com/blog/a-brief-history-of-bureaucrac... [1] https://en.wikipedia.org/wiki/Labyrinth https://en.wikipedia.org/wiki/Labyrinth
    - CreepGin 4y ago
      Wow! I thought it was a typo for "bizarre"... but now I know
    - earleybird 4y ago
      . . . and ducting https://www.youtube.com/watch?v=K9gO01pyv24 https://www.youtube.com/watch?v=K9gO01pyv24
  - malfist 4y ago
    Brazil is Amazon's internal build system. Not sure what GP is talking about here.
    - icedchai 4y ago
      You won't get the reference unless you've seen the movie.
- deleted 4y ago
  [deleted]
- moduspol 4y ago
  If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.
  - saiya-jin 4y ago
    Not necessarily. Probably they already have pool of admins/devops that ran their systems for past 30 years, but have 0 AWS experience. Also, most companies don't need to scale infinitely, not everybody is building next google (in fact, almost nobody outside SV is).
  - Kon-Peki 4y ago
    You can buy on-prem managed storage solutions. Like PureStorage type things, where they call you up if something goes wrong, and are FedEx’ing you replacement parts before you even noticed that it had imperceptibly failed over to the standby power supply or whatever.
  - dijit 4y ago
    There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless you keep things very simple, which cloud providers do not incentivise. One need only look at AWS reference architectures.
    - icedchai 4y ago
      One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.
- neeleshs 4y ago
  The value prop of AWS/GCP etc go beyond bare metal vs VMs. Running HA databases (even if you are a startup, you need this), centralized logging, secrets manager, KMS, making sure disks are encrypted, something like pub/sub as a messaging backbone for your application, load balancers... the list goes on. Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the service on vs deploying, and maintaining a vault on bare machines.
  - MajimasEyepatch 4y ago
    Not to mention compliance. If you have to go through third-party audits (and most businesses do after reaching a certain size), then using standard services from a public cloud provider can simplify things significantly. Sometimes that's because you can just offload "security of the cloud" to the cloud provider, and other times it's because there are well-established guidelines for how to achieve, say, PCI compliance on AWS.
  - Draiken 4y ago
    Yet most of these aren't needed for small crews... do I care that my DB is not HA when I have 1 RPS? You said yes, but I disagree. Do I need centralized logging if I have a few servers? It's already centralized for free. It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services. People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest. You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.
    - neeleshs 4y ago
      YMMV, I run a startup that serves critical customer use cases, and from day 1 I had to care that my DB was HA, backed up, and could be restored. Actually had a case early on where we had to test this because I accidentally deleted customer data and had to restore - took all of one click, with no scripts, no prior investment into backup management etc. We also had to go through infosec reviews from day 1, and many of these are a must have. Doesn't matter if it's a small crew or large. Our customers care. 3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors. Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess

otabdeveloper4 4y ago

Hiring a person to do 'docker compose up' for you is orders of magnitude cheaper than whatever AWS-specific knowldege is needed to not have AWS crap its bed.

boltzmann-brain 4y ago
not just "not have AWS crap its bed", but whatever sacrifices you need to perform in order for amazon not to come back with a bill that just so happens to be $1M higher than you expected it to be because of some gotcha that amazon profits off of, like (say) routing stupid traffic to one of their servers, currently occupied by you, that would be idle otherwise. Nice "mistake" to make.
billythemaniam 4y ago
If their AWS spend is $1M/year, it is not as simple as "docker compose up" on bare metal.
- intelVISA 4y ago
  Might need some systemd as well then to be fair

tester756 4y ago

huge salaries + ten "ops engineers", lol. I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu

citizenpaul 4y ago
One time I was doing research on some really cheap data centers in basically 3rd world countries. Just out of curiosity to see how cheap it could really get. One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture? I thought it was strange they would put that picture on their site and not some GCI or something. I guess at least you know they are not lying about being a real place?
- tester756 4y ago
  I mean, the one I'm talking about is relatively reasonable on infra they've decent building, data center tier, so power/network redundancy, etc, etc they're just cheap as fuck when it comes to people
  - MajimasEyepatch 4y ago
    People are the most important part of any operation! That's the last thing you want to cheap out on.
- choletentent 4y ago
  I’ve just had a good laugh from your comment! Maybe water on the floor was for cooling.
- loloquwowndueo 4y ago
  Share the company name / picture please :)

mbesto 4y ago

Exactly. In the "old IT world" we call this TCO = Total Cost of Ownership.

jjav 4y ago

> have to hire But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away. There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people. At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.

antod 4y ago
Yup, I worked for a company that moved everything from colo to AWS. Not only did the annual tech costs increase about 10x (even with all the AWS migration credit funny math), but they ended up with double the number of engineers to look after it. Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs. The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.
doctor_eval 4y ago
Agreed. I was doing some unrelated research yesterday and stumbled upon the fact that apparently (in Australia) 41% of companies see an increase in IT staff after adopting cloud services. [1] I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves. Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs. [1] https://www2.deloitte.com/au/en/pages/economics/articles/economic-value-cloud-services-australia.html https://www2.deloitte.com/au/en/pages/economics/articles/eco...
- smugma 4y ago

jiveturkey 4y ago

You're not including reliability and availability projections, and the intangible cost of transferable skills wrt infrastructure. (ability to hire sufficiently skilled people to run it)

rstuart4133 4y ago

> If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, I guess you've never done it yourself? I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox. Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live. But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for. 10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.