5 ms·

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For a

So their AWS bill is $500k/year. Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? For an email service, you'd think that you want the most reliable host. Weird.

brenogazzola 4y ago
Their RDS and ES are 500k. They still need to pay for EC2, S3 and any other services they might be using, like ELB and Elasticache. So probably higher than that. He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run? Both your points are still valid though.
- senttoschool 4y ago
  So they'd need to hire a person to setup and manage hardware. And another person to setup and manage RDS, ES, etc. If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k? I don't know. He doesn't provide enough details to make the math work.
  - 988747 4y ago
    $250k is Sillicon Valley rate. In Europe similarly skilled people earn less than that, typically $80k.
    - pmontra 4y ago
      Even $25k if you look at small companies in southern Europe.
      - danwee 4y ago
        That's just for juniors. Seniors are more around $60K-$70K (although, it's still a low number of US standards, yeah)
    - mgdev 4y ago
      37signals pays SV rates for global staff on principle. They publish their salary information, based on job family and job level.
    - deltarholamda 4y ago
      Very important point. There are plenty of skilled people who live in places that don't require that kind of salary in order to afford a decent place to live. Why not hire a guy in Kansas to remote manage your infrastructure? Silicon Valley culture isn't nearly so critical or valuable that your organization requires it in order to do the necessary work.
- datanerd613 4y ago
  He mentioned in the podcast that they rent a space at the data center so I assume colocate
0xblinq 4y ago
Yeah... I don't agree at all with their conclusions about running your own stuff "not that complicated". Sure, it is not "that complicated" for basic stuff, simple sites, hobby projects or overall quick'n dirty solutions because you're under a budget, as with everything. Doing infrastructure properly, with SLAs and a very high level of availability and redundancy (both in systems, and in people with the know-how) is incredibly complicated in my opinion. I bet what they're going to save on the the Cloud provider is going to be spent (and probably even more) in salaries, if they keep the same levels of service. Specially true considering they pay pretty good salaries.
kmos17 4y ago
Managing AWS services also requires staffing, so most likely these would be the same people in a small organization. Self hosting in my experience can be very reliable if you over provision redundant enterprise grade hardware, once it’s running you can get 6-10 years of usable service life. The hardware cost itself is negligible even more so when amortized over a long life cycle.
Nextgrid 4y ago
> Hire one person to buy, configure, manage hardware is what? This point gets raised every single time bare-metal comes up for discussion and at this point I'm not sure if people are legitimately unaware or if there are hidden agendas at play. There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2 - ask for server and get SSH access in minutes/days/weeks (depending if it's an off-the-shelf or custom build). From there, your existing DevOps guy (which you must have to navigate AWS' complexity) can take over. It's only at either extremely large scale or very unusual requirements that managing your own hardware makes sense. Until then, you will never be able to beat the economies of scale of managed bare-metal providers.
- ignoramous 4y ago
  > There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2... I am sorry but how's it different from leaving BigCloud for SmallCloud? That's like saying, I'd no longer use Java/Kotlin because Oracle/Google but I'll however use Clojure because JVM.
  - jstrong 4y ago
    what do you get for $250 a month on ec2? is it 24c/48t dual xeon with 96gb ram, 2x 960gb nvme, with bare metal performance? https://us.ovhcloud.com/bare-metal/infra/infra-4-le/ https://us.ovhcloud.com/bare-metal/infra/infra-4-le/
Kon-Peki 4y ago
> Hire one person Go back and re-read :) They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.
- dijit 4y ago
  Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel. If you're constantly installing stuff then you're scaling up a lot, and that's the level where it would make sense to have someone hired dedicated for it. You can also rent hardware. the thing is: cloud ops are also quite complex but it's a different kind of complexity. I don't know how many people are required. The largest teams I've been on have been cloud deployments, and some of the smallest teams were bare-metal; but that's just an anecdote.
  - deleted 4y ago
    [deleted]
  - Nextgrid 4y ago
    > Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel. If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.
segmondy 4y ago
MSB don't pay $250k/yr. For $100k/yr they will find someone in USA who will enjoy racking servers, wiring, configuring networks, installing OS, etc. If they look outside of US, they will find it for cheap.
PeterStuer 4y ago
If you run anything non trivial in a cloud, chances are that one person will still be cheaper than the one or more person cloud operations team that you have now.
ProAm 4y ago
> Hire one person to buy, configure, manage hardware is what? $250k/year? I think you already pay this guy to manage your AWS instances. so you get the benefit of the full 500k
codegeek 4y ago
"Hire one person to buy, configure, manage hardware is what? $250k/year? " Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.
- Breza 4y ago
  Agreed. Managing servers isn't easy, but it's also not going to get you a quarter million bucks a year.
jjav 4y ago
> So their AWS bill is $500k/year. > Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them). Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server. Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes. You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely. That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.
- wazoox 4y ago
  Also nowadays hardware as well as software are incredibly reliable. You can basically set up a Debian web + postgres server with automatic updates enabled and not need to even log into it for months and months.
  - CoolCold 4y ago
    it would be "interesting" to see such production where DB can be restarted [by autoupdates] on it's own. Should be fine for personal stuff of course.
- Breza 4y ago
  Well said! It's amazing how much power you can get in a $10,000 server. If you have a simple stack and consistent demand, self hosting makes a ton of sense.
pikseladam 4y ago
You don't need to hire.
pelasaco 4y ago
Person + data center, right? Either he will build his own small server room or have to do co-location somewhere..
- Abe9000 4y ago
  WIth 500K yearly you can do a lot!! Including renting 3 office spaces in multiple locations in the US and using them as data centers. Offices cost about $1000 - $3000 yearly, he doesn't need a premium space in a big building but for $500K yearly you can do a lot! In 2 years you would have spent 1M. That is more than enough to have a dedicated devops that travels to all the locations and setup the super beefy servers you can buy with that budget.
  - pelasaco 4y ago
    Ok but 500k is the whole budget for location, backup location, energy, devops people, operation people, etc.. I run part of our infra on-premise, and part of on AWS.. in the summer, we had issues with our air-conditioner (europe) and we had to turn-off partially our service, now in Winter because of the raising costs, we are discussing again what to do with the on-premise services, employees getting ill, etc but AWS, in another hand, is just a bill.
deleted 4y ago
[deleted]

brenogazzola 4y ago

Their RDS and ES are 500k. They still need to pay for EC2, S3 and any other services they might be using, like ELB and Elasticache. So probably higher than that. He also did not mention if they are going to self-host, colocate, or rent bare metal in a more traditional host (like softlayer). Probably colocate as I think that’s how they used to run? Both your points are still valid though.

senttoschool 4y ago
So they'd need to hire a person to setup and manage hardware. And another person to setup and manage RDS, ES, etc. If it's $250k each, that's already $500k. Unless they can find someone who can do both for... $300k? I don't know. He doesn't provide enough details to make the math work.
- 988747 4y ago
  $250k is Sillicon Valley rate. In Europe similarly skilled people earn less than that, typically $80k.
  - pmontra 4y ago
    Even $25k if you look at small companies in southern Europe.
    - danwee 4y ago
      That's just for juniors. Seniors are more around $60K-$70K (although, it's still a low number of US standards, yeah)
  - mgdev 4y ago
    37signals pays SV rates for global staff on principle. They publish their salary information, based on job family and job level.
  - deltarholamda 4y ago
    Very important point. There are plenty of skilled people who live in places that don't require that kind of salary in order to afford a decent place to live. Why not hire a guy in Kansas to remote manage your infrastructure? Silicon Valley culture isn't nearly so critical or valuable that your organization requires it in order to do the necessary work.
datanerd613 4y ago
He mentioned in the podcast that they rent a space at the data center so I assume colocate

0xblinq 4y ago

Yeah... I don't agree at all with their conclusions about running your own stuff "not that complicated". Sure, it is not "that complicated" for basic stuff, simple sites, hobby projects or overall quick'n dirty solutions because you're under a budget, as with everything. Doing infrastructure properly, with SLAs and a very high level of availability and redundancy (both in systems, and in people with the know-how) is incredibly complicated in my opinion. I bet what they're going to save on the the Cloud provider is going to be spent (and probably even more) in salaries, if they keep the same levels of service. Specially true considering they pay pretty good salaries.

kmos17 4y ago

Managing AWS services also requires staffing, so most likely these would be the same people in a small organization. Self hosting in my experience can be very reliable if you over provision redundant enterprise grade hardware, once it’s running you can get 6-10 years of usable service life. The hardware cost itself is negligible even more so when amortized over a long life cycle.

Nextgrid 4y ago

> Hire one person to buy, configure, manage hardware is what? This point gets raised every single time bare-metal comes up for discussion and at this point I'm not sure if people are legitimately unaware or if there are hidden agendas at play. There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2 - ask for server and get SSH access in minutes/days/weeks (depending if it's an off-the-shelf or custom build). From there, your existing DevOps guy (which you must have to navigate AWS' complexity) can take over. It's only at either extremely large scale or very unusual requirements that managing your own hardware makes sense. Until then, you will never be able to beat the economies of scale of managed bare-metal providers.

ignoramous 4y ago
> There are plenty of providers out there who will handle the logistics for you with an interface similar to AWS EC2... I am sorry but how's it different from leaving BigCloud for SmallCloud? That's like saying, I'd no longer use Java/Kotlin because Oracle/Google but I'll however use Clojure because JVM.
- jstrong 4y ago
  what do you get for $250 a month on ec2? is it 24c/48t dual xeon with 96gb ram, 2x 960gb nvme, with bare metal performance? https://us.ovhcloud.com/bare-metal/infra/infra-4-le/ https://us.ovhcloud.com/bare-metal/infra/infra-4-le/

Kon-Peki 4y ago

> Hire one person Go back and re-read :) They [believe that they] don't need to hire anyone. They already have a staff of ops folks that run their cloud operations. And those folks will run their on-prem operations.

dijit 4y ago
Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel. If you're constantly installing stuff then you're scaling up a lot, and that's the level where it would make sense to have someone hired dedicated for it. You can also rent hardware. the thing is: cloud ops are also quite complex but it's a different kind of complexity. I don't know how many people are required. The largest teams I've been on have been cloud deployments, and some of the smallest teams were bare-metal; but that's just an anecdote.
- deleted 4y ago
  [deleted]
- Nextgrid 4y ago
  > Also, Hardware isn't as hard as people make out, you install it once and it runs; if it fails there's a special out-of-band manager built in so you can remotely debug it, datacenters have remote-hands personel. If you're "cloud native" you've already architected your stack to be horizontally-scalable and distributed (partly because the cloud is so expensive that running at peak load capability all the time would bankrupt you) so hardware failure should matter even less.

segmondy 4y ago

MSB don't pay $250k/yr. For $100k/yr they will find someone in USA who will enjoy racking servers, wiring, configuring networks, installing OS, etc. If they look outside of US, they will find it for cheap.

PeterStuer 4y ago

If you run anything non trivial in a cloud, chances are that one person will still be cheaper than the one or more person cloud operations team that you have now.

ProAm 4y ago

> Hire one person to buy, configure, manage hardware is what? $250k/year? I think you already pay this guy to manage your AWS instances. so you get the benefit of the full 500k

codegeek 4y ago

"Hire one person to buy, configure, manage hardware is what? $250k/year? " Outside of expensive Silicon Valley and FAANGs, where do you get this $250k/Year ? I think most of us on HN are normalizing this whole "devs make 250k/year" thing when in reality, it is a tiny percentage of people. Tiny when you look at the entire ecosystem.

Breza 4y ago
Agreed. Managing servers isn't easy, but it's also not going to get you a quarter million bucks a year.

jjav 4y ago

> So their AWS bill is $500k/year. > Hire one person to buy, configure, manage hardware is what? $250k/year? And is this self-hosting more reliable than AWS? A company which has a total AWS bill of $500k/yr is a very small one (the article says 500K is just the RDS/ES bill, so total is surely much higher for them). Our (tiny startup) AWS bill is ~200K/yr and the amount of compute and storage we do could very easily be handled by one single physical server. Now, sure, if we went to self-hosted we'd split that into multiple servers for redundancy and growth headroom, but in any case definitely less than a dozen boxes. You don't hire anyone to install and manage the hardware at that scale. Someone from the ops team goes to the colo for a day or two to set it all up and then you only need to go back when something needs replacement, which at that small scale is going to be very rarely. That's exactly what we did at a previous startup and the company grew to many, many hundreds of people before there was so much infrastructure that it became worth thinking about hiring a dedicated person. By that time you'd be in the 10s of millions in AWS spend.

wazoox 4y ago
Also nowadays hardware as well as software are incredibly reliable. You can basically set up a Debian web + postgres server with automatic updates enabled and not need to even log into it for months and months.
- CoolCold 4y ago
  it would be "interesting" to see such production where DB can be restarted [by autoupdates] on it's own. Should be fine for personal stuff of course.
Breza 4y ago
Well said! It's amazing how much power you can get in a $10,000 server. If you have a simple stack and consistent demand, self hosting makes a ton of sense.

pikseladam 4y ago

You don't need to hire.

pelasaco 4y ago

Person + data center, right? Either he will build his own small server room or have to do co-location somewhere..

Abe9000 4y ago
WIth 500K yearly you can do a lot!! Including renting 3 office spaces in multiple locations in the US and using them as data centers. Offices cost about $1000 - $3000 yearly, he doesn't need a premium space in a big building but for $500K yearly you can do a lot! In 2 years you would have spent 1M. That is more than enough to have a dedicated devops that travels to all the locations and setup the super beefy servers you can buy with that budget.
- pelasaco 4y ago
  Ok but 500k is the whole budget for location, backup location, energy, devops people, operation people, etc.. I run part of our infra on-premise, and part of on AWS.. in the summer, we had issues with our air-conditioner (europe) and we had to turn-off partially our service, now in Winter because of the raising costs, we are discussing again what to do with the on-premise services, employees getting ill, etc but AWS, in another hand, is just a bill.

deleted 4y ago

[deleted]