6 ms·

After that 35% savings, they ended up saving about a US mid level engineer's salary, sans benefits. Hope the time needed for the migration was worth it.

alberth 3y ago
I'm sure they are also getting better performance as well. Not sure how to factor that $ into the equation.
- flkenosad 3y ago
  Also, I'd imagine most companies can fill unused compute with long-running batch jobs so you're getting way more bang for your buck. It's really egregious what these clouds are charging.
darkwater 3y ago
To get real savings with a complex enough project you will need one or more FTE salaries just to stay on top of AWS spending optimizations
- baz00 3y ago
  Plus... 2x FTEs to manage the AWS support tickets 3x FTE to understand the differences between the AWS bundled products and open source stuff which you can't get close enough to the config for so that you can actually use it as intended. 3x Security folk to work out how to manage the tangle of multiple accounts, networks, WAF and compliance overheads 3x FTEs to write HCL and YAML to support the cloud. 2x Solution architects to try and rebuild everything cloud native and get stuck in some technicality inside step functions for 9 months and achieve nothing. 1x extra manager to sit in meetings with AWS once a week and bitch about the crap support, the broken OSS bundled stuff and work out weird network issues. 1x cloud janitor to clean up all the dirt left around the cluster burning cash. --- Footnote: Was this to free us or enslave us?
  - hotpotamus 3y ago
    > Footnote: Was this to free us or enslave us? I assume whichever provides more margin to Jeff Bezos.
  - gymbeaux 3y ago
    Our experience hasn’t been THAT bad but we did waste a lot of time in weekly meetings with AWS “solutions architects” who knew next to nothing about AWS aside from a shallow, salesman-like understanding. They make around $150k too, by the way. I tried to apply to be one, but AWS wants someone with more sales experience and they don’t really care about my AWS certs
    - baz00 3y ago
      As an AWS Solution Architect (independent untethered to Bezos) I resent that comment. I know slightly more than next to nothing about AWS and I can Google something and come up with something convincing and sell it to you in a couple of minutes!
      - gymbeaux 3y ago
        How do I make $150k (or more) having high-level conversations about AWS with senior software engineers? Seriously. I’m sure it’s not an “easy” job, but fuck if I make less actually writing the software (median SWE salary is something like $140k in the US- depends on who you ask but it’s not the $250k+ that Levels.fyi would lead you to believe).
    - scarface_74 3y ago
      I can guarantee an SA working for AWS makes more than $150k. A returning L4 intern makes that much (former AWS Professional Services employee) And no one cares about AWS certifications. They are proof of nothing and disregarded by anyone with a modicum of a clue. I’m speaking as someone who once had nine active certifications and I believe I still have six active ones. I only got them as a guided learning path. I knew going in they were meaningless.
makeitdouble 3y ago
Getting a bare metal stack has interesting side effects on how they can plan future projects. One that's not immediately obvious is to keep on staff experienced infra engineers that bring their expertise for designing future projects. Another is the option to tackle project in ways that would be to costly if they were still on AWS (e.g. ML training, stuff with long and heavy CPU load).
- flkenosad 3y ago
  Yep and hardware is only getting cheaper. Better to just buy more drives/chips when you need them.
  - jbverschoor 3y ago
    M4.xlarge (m4.xlarge, 4cpu, 16gb, no storage) is about $100 per month. This cpu is 7 years old(slow! and power hungry) and has 36 threads. This means it runs 18 of these instances. The total revenue so far for one cpu is 100x18x12x7 = $150k If used as a spot instance it’s 144/month, so about 200k A standard i9-14700k gen has 32 threads, but it can run 12 of these instances (max 192mem). This CPU will cost you $800. Memory is cheap, so for about 1-2k you’re all set, and have a machine that’s way faster and cheaper. Basically, buy a bunch of NUCs and you’re saving yourself around $1500 per month per NUC. It pays itself back in 1 month Cloud hosting is —insane— Not even touching memory ballooning for mostly idle applications. Lastly, don’t give me reliability s an argument. These were all ephemeral instances that have no storage, so you’ll have to pay for that slow non-nvme storage platform.
    - mr_toad 3y ago
      If you can do it for some much less than Amazon (and all the other cloud vendors), then why don’t you create your own cloud and undercut them?
      - peteradio 3y ago
        Hosting for the world is different than hosting for yourself.
efitz 3y ago
I was thinking the same thing. If the migration took more than one man-year then they lost money. Also what happens at hardware end-of-life? Also what happens if they encounter an explosive growth or burst usage event? And did their current staffing include enough headcount to maintain the physical machines or did they have to hire for that? Etc etc. Cloud is not cheap but if you are honest about TCO then the savings likely are WAY less than they imply in the article.
- flkenosad 3y ago
  > If the migration took more than one man-year then they lost money. Your math is incorrect. The savings are per year. The job gets done once. > Also what happens at hardware end-of-life? You buy more hardware. A drive should last a few years on average at least. > Also what happens if they encounter an explosive growth or burst usage event? Short term, clouds are always available to handle extra compute. It's not a bad idea to use a cloud load-balancing system anyway to handle spam or caching. But also, you can buy hardware from amazon and get it the next day with Prime. > And did their current staffing include enough headcount to maintain the physical machines or did they have to hire for that? I'm sure any team capable of building complex software at scale is capable of running a few servers on prem. I'm sure there's more than a few programmers on most teams that have homelabs they muck around with. > Etc etc. I'd love to hear more arguments.
  - m00x 3y ago
    The job is never done once. Not in hardware.
- sgarland 3y ago
  > Also what happens if they encounter an explosive growth or burst usage event? TFA states that they maintain their AWS account, and can spin up additional compute in ~10 minutes.
awslol 3y ago
They also saved the salaries of the team whose job was doing nothing but chasing misplaced spaces in yaml configuration files. Cloud infrastructure doesn't just appear out of thin air. You have to hire people to describe what you want to do. And with the complexity mess we're in today it's not at all clear which takes more effort.
- icedchai 3y ago
  To manage a large fleet of physical servers, you need similar ops skills. You're not going to configure all those systems by hand, are you?
  - awslol 3y ago
    Depends on the size of the fleet. If you're using less than a dozen servers manual configuration is simpler. Depending on what you're doing that could mean serving a hundred million customers. Which is plenty for most business.
    - icedchai 3y ago
      A dozen servers would be pushing it. It's not the size of the fleet, it's the consistency and repeatability of configuration and deployment. I assume there are other, non-production servers: various dev environments, test/QA. How do you ensure every environment is built to spec without some level of automation? It's the old "pets" vs "cattle" argument. I've worked at companies with their own data centers and manual configuration. Every system was a pet.
  - postalrat 3y ago
    They spent $150,000 on physical servers. Probably 1 or 2 racks. Not much of a 'fleet'.
    - ketzo 3y ago
      I mean, “fleet” semantics aside, surely $150,000 of servers is enough for at least one full-time person to be maintaining them. The point is that there are absolutely maintenance and ops costs associated with these servers.
dilyevsky 3y ago
I broke the rules and read the article first: > In the context of AWS, the expenses associated with employing AWS administrators often exceed those of Linux on-premises server administrators. This represents an additional cost-saving benefit when shifting to bare metal. With today’s servers being both efficient and reliable, the need for “management” has significantly decreased. I also never seen an eng org where substantial part of it didn’t do useless projects that never amount to anything
- rewmie 3y ago
  I get the point that they tried to make, but this comparison between "AWS administrators" and "Linux on-premises server administrators" is beyond apple-and-oranges and is actually completely meaningless. A team does not use AWS because it provides compute. AWS, even when using barebonea EC2 instances, actually means on-demand provisioning of computational resources with the help of infrastructure-as-code services. A random developer logs into his AWS console, clicks a few buttons, and he's already running a fully instrumented service with logging and metrics a click away. He can click another button and delete/shut down everything. He can click on a button again and deploy the same application in multiple continents with static files provided through a global CDN, deployed with a dedicated pipeline. He clicks on another button again and everything is shut down again. How do you pull that off with "Linux on-premises server administrators"? You don't. At most, you can get your Linux server administrators to manage their hardware with something like OpenStack, but they would be playing the role of the AWS engineers that your "AWS administrators" don't even know exist. However, anyone who works with AWS only works on the abstraction layers above that which a "Linux on premises administrator" works on.
  - gnaritas99 3y ago
    [dead]
  - dilyevsky 3y ago
    > A random developer logs into his AWS console, clicks a few buttons, and he's already running a fully instrumented service with logging and metrics a click away... This only works that way for very small spend orgs that haven’t implemented soc 2 or the like. If that’s what you’re doing then probably should stay away from datacenter, sure
dheera 3y ago
They probably need to now hire 24/7 security to watch the bare metal if they're serious about it, so not sure about that engineer
- dilyevsky 3y ago
  onsite security is offered by the colo provider. You can also pay for locked cabinets with cameras and anti-tampering or even completely caged off depending on your security requirements
baz00 3y ago
If we saved 35% that could hire 20 FTEs. Not that we'd need them as we wouldn't have to write as much HCL.
stcroixx 3y ago
Freedom from vendor lock in is hard to put a value on, but for me it’s definitely worth a mid level engineers salary in any context.
- scarface_74 3y ago
  Yes, as long as you ignore the other literally 100 or so SaaS products that the average enterprise uses. You are always locked in to your infrastructure at any decent scale
avereveard 3y ago
And the servers are brand new, so maintenance is not factor in yet.
jcelerier 3y ago
outside of the US you could likely pay for two mid or three juniors for this though, e.g. in France a junior engineer in an average city would likely be around 46-48k$USD for the total employer cost and France is already expensive compared to a lot of other countries with talented engineers

alberth 3y ago

I'm sure they are also getting better performance as well. Not sure how to factor that $ into the equation.

flkenosad 3y ago
Also, I'd imagine most companies can fill unused compute with long-running batch jobs so you're getting way more bang for your buck. It's really egregious what these clouds are charging.

darkwater 3y ago

To get real savings with a complex enough project you will need one or more FTE salaries just to stay on top of AWS spending optimizations

baz00 3y ago
Plus... 2x FTEs to manage the AWS support tickets 3x FTE to understand the differences between the AWS bundled products and open source stuff which you can't get close enough to the config for so that you can actually use it as intended. 3x Security folk to work out how to manage the tangle of multiple accounts, networks, WAF and compliance overheads 3x FTEs to write HCL and YAML to support the cloud. 2x Solution architects to try and rebuild everything cloud native and get stuck in some technicality inside step functions for 9 months and achieve nothing. 1x extra manager to sit in meetings with AWS once a week and bitch about the crap support, the broken OSS bundled stuff and work out weird network issues. 1x cloud janitor to clean up all the dirt left around the cluster burning cash. --- Footnote: Was this to free us or enslave us?
- hotpotamus 3y ago
  > Footnote: Was this to free us or enslave us? I assume whichever provides more margin to Jeff Bezos.
- gymbeaux 3y ago
  Our experience hasn’t been THAT bad but we did waste a lot of time in weekly meetings with AWS “solutions architects” who knew next to nothing about AWS aside from a shallow, salesman-like understanding. They make around $150k too, by the way. I tried to apply to be one, but AWS wants someone with more sales experience and they don’t really care about my AWS certs
  - baz00 3y ago
    As an AWS Solution Architect (independent untethered to Bezos) I resent that comment. I know slightly more than next to nothing about AWS and I can Google something and come up with something convincing and sell it to you in a couple of minutes!
    - gymbeaux 3y ago
      How do I make $150k (or more) having high-level conversations about AWS with senior software engineers? Seriously. I’m sure it’s not an “easy” job, but fuck if I make less actually writing the software (median SWE salary is something like $140k in the US- depends on who you ask but it’s not the $250k+ that Levels.fyi would lead you to believe).
  - scarface_74 3y ago
    I can guarantee an SA working for AWS makes more than $150k. A returning L4 intern makes that much (former AWS Professional Services employee) And no one cares about AWS certifications. They are proof of nothing and disregarded by anyone with a modicum of a clue. I’m speaking as someone who once had nine active certifications and I believe I still have six active ones. I only got them as a guided learning path. I knew going in they were meaningless.

makeitdouble 3y ago

Getting a bare metal stack has interesting side effects on how they can plan future projects. One that's not immediately obvious is to keep on staff experienced infra engineers that bring their expertise for designing future projects. Another is the option to tackle project in ways that would be to costly if they were still on AWS (e.g. ML training, stuff with long and heavy CPU load).

flkenosad 3y ago
Yep and hardware is only getting cheaper. Better to just buy more drives/chips when you need them.
- jbverschoor 3y ago
  M4.xlarge (m4.xlarge, 4cpu, 16gb, no storage) is about $100 per month. This cpu is 7 years old(slow! and power hungry) and has 36 threads. This means it runs 18 of these instances. The total revenue so far for one cpu is 100x18x12x7 = $150k If used as a spot instance it’s 144/month, so about 200k A standard i9-14700k gen has 32 threads, but it can run 12 of these instances (max 192mem). This CPU will cost you $800. Memory is cheap, so for about 1-2k you’re all set, and have a machine that’s way faster and cheaper. Basically, buy a bunch of NUCs and you’re saving yourself around $1500 per month per NUC. It pays itself back in 1 month Cloud hosting is —insane— Not even touching memory ballooning for mostly idle applications. Lastly, don’t give me reliability s an argument. These were all ephemeral instances that have no storage, so you’ll have to pay for that slow non-nvme storage platform.
  - mr_toad 3y ago
    If you can do it for some much less than Amazon (and all the other cloud vendors), then why don’t you create your own cloud and undercut them?
    - peteradio 3y ago
      Hosting for the world is different than hosting for yourself.

efitz 3y ago

I was thinking the same thing. If the migration took more than one man-year then they lost money. Also what happens at hardware end-of-life? Also what happens if they encounter an explosive growth or burst usage event? And did their current staffing include enough headcount to maintain the physical machines or did they have to hire for that? Etc etc. Cloud is not cheap but if you are honest about TCO then the savings likely are WAY less than they imply in the article.

flkenosad 3y ago
> If the migration took more than one man-year then they lost money. Your math is incorrect. The savings are per year. The job gets done once. > Also what happens at hardware end-of-life? You buy more hardware. A drive should last a few years on average at least. > Also what happens if they encounter an explosive growth or burst usage event? Short term, clouds are always available to handle extra compute. It's not a bad idea to use a cloud load-balancing system anyway to handle spam or caching. But also, you can buy hardware from amazon and get it the next day with Prime. > And did their current staffing include enough headcount to maintain the physical machines or did they have to hire for that? I'm sure any team capable of building complex software at scale is capable of running a few servers on prem. I'm sure there's more than a few programmers on most teams that have homelabs they muck around with. > Etc etc. I'd love to hear more arguments.
- m00x 3y ago
  The job is never done once. Not in hardware.
sgarland 3y ago
> Also what happens if they encounter an explosive growth or burst usage event? TFA states that they maintain their AWS account, and can spin up additional compute in ~10 minutes.

awslol 3y ago

They also saved the salaries of the team whose job was doing nothing but chasing misplaced spaces in yaml configuration files. Cloud infrastructure doesn't just appear out of thin air. You have to hire people to describe what you want to do. And with the complexity mess we're in today it's not at all clear which takes more effort.

icedchai 3y ago
To manage a large fleet of physical servers, you need similar ops skills. You're not going to configure all those systems by hand, are you?
- awslol 3y ago
  Depends on the size of the fleet. If you're using less than a dozen servers manual configuration is simpler. Depending on what you're doing that could mean serving a hundred million customers. Which is plenty for most business.
  - icedchai 3y ago
    A dozen servers would be pushing it. It's not the size of the fleet, it's the consistency and repeatability of configuration and deployment. I assume there are other, non-production servers: various dev environments, test/QA. How do you ensure every environment is built to spec without some level of automation? It's the old "pets" vs "cattle" argument. I've worked at companies with their own data centers and manual configuration. Every system was a pet.
- postalrat 3y ago
  They spent $150,000 on physical servers. Probably 1 or 2 racks. Not much of a 'fleet'.
  - ketzo 3y ago
    I mean, “fleet” semantics aside, surely $150,000 of servers is enough for at least one full-time person to be maintaining them. The point is that there are absolutely maintenance and ops costs associated with these servers.

dilyevsky 3y ago

I broke the rules and read the article first: > In the context of AWS, the expenses associated with employing AWS administrators often exceed those of Linux on-premises server administrators. This represents an additional cost-saving benefit when shifting to bare metal. With today’s servers being both efficient and reliable, the need for “management” has significantly decreased. I also never seen an eng org where substantial part of it didn’t do useless projects that never amount to anything

rewmie 3y ago
I get the point that they tried to make, but this comparison between "AWS administrators" and "Linux on-premises server administrators" is beyond apple-and-oranges and is actually completely meaningless. A team does not use AWS because it provides compute. AWS, even when using barebonea EC2 instances, actually means on-demand provisioning of computational resources with the help of infrastructure-as-code services. A random developer logs into his AWS console, clicks a few buttons, and he's already running a fully instrumented service with logging and metrics a click away. He can click another button and delete/shut down everything. He can click on a button again and deploy the same application in multiple continents with static files provided through a global CDN, deployed with a dedicated pipeline. He clicks on another button again and everything is shut down again. How do you pull that off with "Linux on-premises server administrators"? You don't. At most, you can get your Linux server administrators to manage their hardware with something like OpenStack, but they would be playing the role of the AWS engineers that your "AWS administrators" don't even know exist. However, anyone who works with AWS only works on the abstraction layers above that which a "Linux on premises administrator" works on.
- gnaritas99 3y ago
  [dead]
- dilyevsky 3y ago
  > A random developer logs into his AWS console, clicks a few buttons, and he's already running a fully instrumented service with logging and metrics a click away... This only works that way for very small spend orgs that haven’t implemented soc 2 or the like. If that’s what you’re doing then probably should stay away from datacenter, sure

dheera 3y ago

They probably need to now hire 24/7 security to watch the bare metal if they're serious about it, so not sure about that engineer

dilyevsky 3y ago
onsite security is offered by the colo provider. You can also pay for locked cabinets with cameras and anti-tampering or even completely caged off depending on your security requirements

baz00 3y ago

If we saved 35% that could hire 20 FTEs. Not that we'd need them as we wouldn't have to write as much HCL.

stcroixx 3y ago

Freedom from vendor lock in is hard to put a value on, but for me it’s definitely worth a mid level engineers salary in any context.

scarface_74 3y ago
Yes, as long as you ignore the other literally 100 or so SaaS products that the average enterprise uses. You are always locked in to your infrastructure at any decent scale

avereveard 3y ago

And the servers are brand new, so maintenance is not factor in yet.

jcelerier 3y ago

outside of the US you could likely pay for two mid or three juniors for this though, e.g. in France a junior engineer in an average city would likely be around 46-48k$USD for the total employer cost and France is already expensive compared to a lot of other countries with talented engineers