4 ms·

We've [1] been using Hetzner's dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we t

We've [1] been using Hetzner's dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we typically see request times half. And because the hardware is cheaper we can provide dedicated DevOps engineering time to each client. There are some caveats though: 1) A staging cluster for testing updates is really a must. YOLO-ing prod updates on a Sunday is no one's idea of fun. 2) Application level replication is king, followed by block-level replication (we use OpenEBS/Mayastor). After going through all the Postgres operators we found StackGres to (currently) be the best. 3) The Ansible playbooks are your assets. Once you have them down and well-commented for a given service then re-deploying that service in other cases (or again in the future) becomes straightforward. 4) If you can I'd recommend a dedicated 10G network to connect your servers. 1G just isn't quite enough when it comes to the combined load of prod traffic, plus image pulls, plus inter-service traffic. This also gives a 10x latency improvement over AWS intra-az. 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). 6) MinIO clusters (via the operator) are not that tricky to operate as long as you follow the well trodden path. This provides you with local high-bandwidth, low-latency object storage. 7) The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time. 8) You can still push ancillary/annoying tasks off onto cloud providers (personally I'm a fan of CloudFlare for HTTP load balancing). [1]: https://lithus.eu https://lithus.eu

sureIy 2y ago
> I'd put it at 2-4 months of undistracted skilled engineering time. How much is that worth to your company/customer vs a higher monthly bill for the next 5 years? As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all, unless I expect a 10k/month AWS bill. xkcd comes to mind: https://xkcd.com/1319/ https://xkcd.com/1319/
- adamcharnock 2y ago
  > As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all. Well I do rather agree, but as a consultancy I'm biased. But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers). Say one wants a 3 year payback, that is $2,200/month savings you need. Which seems highly achievable given some of the cloud spends I've seen, and that I think an 80-90% reduction in cloud spend is a good ballpark. The appeal of a consultancy is that we'll remove the up-front investment, provide the skills, de-risk the whole endeavour, even put engineers within your team, but you'll _only_ save 50%. The latter option is much more appealing in terms of hiring, risk, and cash-flow. But if your company has the skills, the cash, and the risk tolerance then maybe the former approach is best. EDIT: I actually think the(/our) consultancy option is a really good idea for startups. Their infrastructure ends up being slightly over-built to start with, but very quickly they end up saving a lot of money, and they also get DevOps staffing without having to hire for it. Moreover, the DevOps resource available to them scales with their compute needs. (also we offer 2x the amount of DevOps days for startups for the first year to help them get up and running).
  - nkmnz 2y ago
    this assumes there are no devops/consulting cost to setup something with AWS. My experience is that "the aws way of doing XYZ" is almost as complicated as doing it the non-AWS-way. On top of that: the non-AWS-way is much more portable across hosting providers, so you decrease your business risks considerably.
    - adamcharnock 2y ago
      I wholeheartedly agree, I'm trying to be generous as I know I have a bias here. I think the AWS way made clear sense in the days before the current generation of tooling existed, when we were SSH-ing into our snowflake servers (for example). But now we have tools like Kubernetes/Nomad/OpenShift/etc/etc, the logic just doesn't seem to add up any more. The main argument against it is generally of the form, "Yes, but we don't want to hire for non-cloud/bare-metal". Which is why I think a consultancy provides a good middle ground here – trading off cost savings against business factors.
bigbones 2y ago
> dedicated 10G network to connect your servers Do you have to ask Hetzner nicely for this? They have a publicly documented 10G uplink option, but that is for external networking and IMHO heavily limited (20TB limit). For internal cluster IO 20TB could easily become a problem
- nh2 2y ago
  Hetzner does not charge for internal bandwidth.
- adamcharnock 2y ago
  It is under their costing for 'additional hardware'[1]. You need to factor in the switch, uplink for each server, and the NIC for each server. [1]: https://docs.hetzner.com/robot/general/pricing/price-list-for-additional-products/ https://docs.hetzner.com/robot/general/pricing/price-list-fo...
bambambazooka 2y ago
> 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). Are you willing to share example config for that part?
- adamcharnock 2y ago
  I don't have one I can share publicly, but if you send me an email I'll see what I can do :-) Email is in my profile. You'll need a bit of baseline networking knowledge.
  - bc569a80a344f9c 2y ago
    Should note that if you don't have enough networking knowledge, this is an excellent way to build a gun to shoot yourself in the foot with. If you misconfigure BGP or don't take basic precautions such as sanity filters on in- and outbound routes, you can easily do something silly like overwrite each server's default route, taking down all your services. It's not rocket science, but it is complex, and building something complex you don't fully understand for production services can be a very bad idea.
lucasrattz 2y ago
> The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time. Perhaps you could take a look at https://syself.com https://syself.com (Disclaimer: I'm an employee there). We built a platform that gives you production-ready clusters in a few minutes.

sureIy 2y ago

> I'd put it at 2-4 months of undistracted skilled engineering time. How much is that worth to your company/customer vs a higher monthly bill for the next 5 years? As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all, unless I expect a 10k/month AWS bill. xkcd comes to mind: https://xkcd.com/1319/ https://xkcd.com/1319/

adamcharnock 2y ago
> As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all. Well I do rather agree, but as a consultancy I'm biased. But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers). Say one wants a 3 year payback, that is $2,200/month savings you need. Which seems highly achievable given some of the cloud spends I've seen, and that I think an 80-90% reduction in cloud spend is a good ballpark. The appeal of a consultancy is that we'll remove the up-front investment, provide the skills, de-risk the whole endeavour, even put engineers within your team, but you'll _only_ save 50%. The latter option is much more appealing in terms of hiring, risk, and cash-flow. But if your company has the skills, the cash, and the risk tolerance then maybe the former approach is best. EDIT: I actually think the(/our) consultancy option is a really good idea for startups. Their infrastructure ends up being slightly over-built to start with, but very quickly they end up saving a lot of money, and they also get DevOps staffing without having to hire for it. Moreover, the DevOps resource available to them scales with their compute needs. (also we offer 2x the amount of DevOps days for startups for the first year to help them get up and running).
- nkmnz 2y ago
  this assumes there are no devops/consulting cost to setup something with AWS. My experience is that "the aws way of doing XYZ" is almost as complicated as doing it the non-AWS-way. On top of that: the non-AWS-way is much more portable across hosting providers, so you decrease your business risks considerably.
  - adamcharnock 2y ago
    I wholeheartedly agree, I'm trying to be generous as I know I have a bias here. I think the AWS way made clear sense in the days before the current generation of tooling existed, when we were SSH-ing into our snowflake servers (for example). But now we have tools like Kubernetes/Nomad/OpenShift/etc/etc, the logic just doesn't seem to add up any more. The main argument against it is generally of the form, "Yes, but we don't want to hire for non-cloud/bare-metal". Which is why I think a consultancy provides a good middle ground here – trading off cost savings against business factors.

bigbones 2y ago

> dedicated 10G network to connect your servers Do you have to ask Hetzner nicely for this? They have a publicly documented 10G uplink option, but that is for external networking and IMHO heavily limited (20TB limit). For internal cluster IO 20TB could easily become a problem

nh2 2y ago
Hetzner does not charge for internal bandwidth.
adamcharnock 2y ago
It is under their costing for 'additional hardware'[1]. You need to factor in the switch, uplink for each server, and the NIC for each server. [1]: https://docs.hetzner.com/robot/general/pricing/price-list-for-additional-products/ https://docs.hetzner.com/robot/general/pricing/price-list-fo...

bambambazooka 2y ago

> 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). Are you willing to share example config for that part?

adamcharnock 2y ago
I don't have one I can share publicly, but if you send me an email I'll see what I can do :-) Email is in my profile. You'll need a bit of baseline networking knowledge.
- bc569a80a344f9c 2y ago
  Should note that if you don't have enough networking knowledge, this is an excellent way to build a gun to shoot yourself in the foot with. If you misconfigure BGP or don't take basic precautions such as sanity filters on in- and outbound routes, you can easily do something silly like overwrite each server's default route, taking down all your services. It's not rocket science, but it is complex, and building something complex you don't fully understand for production services can be a very bad idea.

lucasrattz 2y ago

> The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time. Perhaps you could take a look at https://syself.com https://syself.com (Disclaimer: I'm an employee there). We built a platform that gives you production-ready clusters in a few minutes.