5 ms·

Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access

Would be interested to know what your hosting bill was like? We recently moved everything in-house and started hosting our own servers. Having physical access to the machine (and the network HW) pretty much eliminated all of our dev ops (1). We expected the HW to fail often and such, but in 2 years the only maintenance we had to do was add more disks to the RAID array and resize the partition a few times. Even though we have redundant everything, nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two). (1): How? Well instead of running small containers/micro boxes/services that we could scale up and down, since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore. We don't really care how much we utilize our own HW, so we just smooshed everything onto huge VMs in two huge 100 core boxes with affectionate names. We used super basic networking features (VLANs, firewall port forward) to setup the network. To simplify dev ops even further, there is only access to the servers from specific ethernet ports in the building.

olive247 3y ago
Does geolocation of the server not matter for your application? I am curious if there are any latency issues with this solution as opposed to a hosted option.
ghiculescu 3y ago
It's around 150k USD/month at the moment. Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.
- aetherspawn 3y ago
  Our deployment is much smaller than yours! Our upfront hardware cost including carpentry and HVAC was around $80k, our monthly cost in cloud was around $5k/mo and now we pay only about $1k/mo in energy (which includes the existing $500/mo bill from the office). We could reduce this further with solar panels, but we share the tall-ish building with 13 other suites and we couldn't get approval to take up most of the roof.
- Sai_ 3y ago
  fyi - tanda.co gives a 403 forbidden error
- martinald 3y ago
  What does spend break down as?
- welder 3y ago
  Holy shit that's more than I expected for your type of business. I'm running a SaaS too with half a million users and a lot of data processing, but my hosting costs are way lower... under $10k/mo
Buttons840 3y ago
> We expected the HW to fail often and such, but in 2 years ... nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two). I began my career by scripting Asterisk using Python. I created a daemon using Twisted and that process ran for 3 years without fail (as in, the same Linux OS process was running for 3 years). This was on commodity hardware.
- deleted 3y ago
  [deleted]
withinboredom 3y ago
IME, if a drive is going to fail, it's going to fail in the first hundred hours or so of use. In my entire life (of using hundreds of disks), I've only had three failures. Two of those were on the same server, at the same time, in the same RAID array and took out the whole array -- the data was gone forever. Thank the gods for backups. So, just make sure you take backups pretty regularly because you never know.
8organicbits 3y ago
> since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually. [1] https://instances.vantage.sh/aws/ec2/c6a.32xlarge https://instances.vantage.sh/aws/ec2/c6a.32xlarge [2] https://www.hetzner.com/dedicated-rootserver/rx170 https://www.hetzner.com/dedicated-rootserver/rx170
- cjalmeida 3y ago
  This is the way. Unless you are huge, self hosting and even colo is not competitive with long term VM commitments in hyper scalers. They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.
- aetherspawn 3y ago
  One of our 100 core boxes is Windows Server and it looks like they want $79,949 annually for it.
  - aetherspawn 3y ago
    I wanted to add more info but couldn't edit. We bought that server for $12,000 with 5 year warranty, and got our Windows Server licenses out of an MS bundle which cost about $400. I think that brings the ROI after 5 years compared with AWS to something like 3200%, less energy, if nothing goes wrong. (I know that there are better options than running Windows Server on this box, but it seems to perform the same, it's far easier to administer by inexperienced devops, and in the long term it's actually around $1500/yr cheaper than running supported RHEL. Our other box is running supported RHEL but after testing Windows Server and having no issues we'll probably switch it for the cost savings).
- riku_iki 3y ago
  > On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. plus more expensive network for cloud

olive247 3y ago

Does geolocation of the server not matter for your application? I am curious if there are any latency issues with this solution as opposed to a hosted option.

ghiculescu 3y ago

It's around 150k USD/month at the moment. Would love to learn more about your setup. Email me if you're interested in chatting. Email in my profile.

aetherspawn 3y ago
Our deployment is much smaller than yours! Our upfront hardware cost including carpentry and HVAC was around $80k, our monthly cost in cloud was around $5k/mo and now we pay only about $1k/mo in energy (which includes the existing $500/mo bill from the office). We could reduce this further with solar panels, but we share the tall-ish building with 13 other suites and we couldn't get approval to take up most of the roof.
Sai_ 3y ago
fyi - tanda.co gives a 403 forbidden error
martinald 3y ago
What does spend break down as?
welder 3y ago
Holy shit that's more than I expected for your type of business. I'm running a SaaS too with half a million users and a lot of data processing, but my hosting costs are way lower... under $10k/mo

Buttons840 3y ago

> We expected the HW to fail often and such, but in 2 years ... nothing has ever failed and the server has never gone offline (except when it reboots every Sunday @ 3AM in a min or two). I began my career by scripting Asterisk using Python. I created a daemon using Twisted and that process ran for 3 years without fail (as in, the same Linux OS process was running for 3 years). This was on commodity hardware.

deleted 3y ago
[deleted]

withinboredom 3y ago

IME, if a drive is going to fail, it's going to fail in the first hundred hours or so of use. In my entire life (of using hundreds of disks), I've only had three failures. Two of those were on the same server, at the same time, in the same RAID array and took out the whole array -- the data was gone forever. Thank the gods for backups. So, just make sure you take backups pretty regularly because you never know.

8organicbits 3y ago

> since we owned the hardware it meant that efficient utilization wasn't a requirement for us anymore This is a great point. Hyper focus on optimizing micro-containers is costly and sucks up a lot of time. You can still deploy like this in the cloud though. On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. Hetzner is cheaper, 80 vCPU for about €2,506 annually. [1] https://instances.vantage.sh/aws/ec2/c6a.32xlarge https://instances.vantage.sh/aws/ec2/c6a.32xlarge [2] https://www.hetzner.com/dedicated-rootserver/rx170 https://www.hetzner.com/dedicated-rootserver/rx170

cjalmeida 3y ago
This is the way. Unless you are huge, self hosting and even colo is not competitive with long term VM commitments in hyper scalers. They get big discounts on everything you'd pay retail, and their margins for barebones VM is not that large.
aetherspawn 3y ago
One of our 100 core boxes is Windows Server and it looks like they want $79,949 annually for it.
- aetherspawn 3y ago
  I wanted to add more info but couldn't edit. We bought that server for $12,000 with 5 year warranty, and got our Windows Server licenses out of an MS bundle which cost about $400. I think that brings the ROI after 5 years compared with AWS to something like 3200%, less energy, if nothing goes wrong. (I know that there are better options than running Windows Server on this box, but it seems to perform the same, it's far easier to administer by inexperienced devops, and in the long term it's actually around $1500/yr cheaper than running supported RHEL. Our other box is running supported RHEL but after testing Windows Server and having no issues we'll probably switch it for the cost savings).
riku_iki 3y ago
> On AWS make a three-year reservation of a c6a.32xlarge to get 128 vCPU for $16k annually. plus more expensive network for cloud