15 ms·

Kubernetes on Hetzner: cutting my infra bill by 75%

375 points by BillFranklin 2y ago

deleted 2y ago
[deleted]
dvfjsdhgfv 2y ago
> Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner's volumes we were seeing >50ms of IOWAIT with very low IOPS. There is a surprisingly easy way to address this issue: use (ridiculously cheap) Hetzner metal machines as nodes. The ones with nvme storage offer excellent performance for dbs and often have generous amounts of RAM. I'd go as far as to say you'd be better off to invest in two or more beefy bare metal machines for a master-replica(s) setup rather than run the db on k8s. If you don't want to be bothered with the setup, you can use one of many modern packages such as Pigsty: https://pigsty.cc/ https://pigsty.cc/ (not affiliated but a huge fan).
- BillFranklin 2y ago
  Thanks, hadn’t heard of pigsty. As you say, I had to use nvme ssds for the dbs, the performance is pretty good so I didn’t look to get metal nodes.
- threeseed 2y ago
  There are plenty of options for running a database on Kubernetes whilst using local NVMe storage. There are just pinning the database pods to specific nodes and using a LocalPathProvisioner or distributed solutions like JuiceFS, OpenEBS etc.
- gourneau 2y ago
  Thanks for the Pigsty link. I have been a big fan of running Postgres on metal machines.
- lucasrattz 2y ago
  I've had great experiences with using the bare metal server's local storage. This is the guide I wrote for our customers: https://syself.com/docs/hetzner/apalla/how-to-guides/storage/local-storage-in-baremetal https://syself.com/docs/hetzner/apalla/how-to-guides/storage...
cjr 2y ago
What about cluster autoscaling?
- BillFranklin 2y ago
  I didn’t touch on that in the article, but essentially it’s a one line change to add a worker node (or nodes) to the cluster, then it’s automatically enrolled. We don’t have such bursty requirements fortunately so I have not needed to automate this.
- preisschild 2y ago
  Works rather well. I use CAPI + Cluster-Autoscaler + Talos and new nodes are provisioned and ready within 2-3 minutes.
postepowanieadm 2y ago
Lovely website.
segmondy 2y ago
Great write up Bill!
aliasxneo 2y ago
I’m planning on doing something similar but want to use Talos with bare metal machines. I suspect to see similar price reductions from our current EKS bill.
- threeseed 2y ago
  Depending on your cluster size I highly recommend Omni: https://omni.siderolabs.com https://omni.siderolabs.com It took minutes to setup a cluster and I love having a UI to see what is happening. I wish there were more products like this as I suspect there will be a trend towards more self-managed Kubernetes clusters given how expensive the cloud is becoming.
- MathiasPius 2y ago
  I set up a Talos bare metal cluster about a year ago, and documented the whole process on my website. Feel free to reach out if you have any questions!
  - cedws 2y ago
    Any thoughts/feelings about Talos vs Bottlerocket?
    - p_l 2y ago
      Did bottlerocket get any closer to stable and usable outside AWS walled garden? Last time I tried was admittedly in 2022, but in testing which distro to go with bottlerocket lost because we couldn't setup local builds...
    - MathiasPius 2y ago
      I've only used Bottlerocket in relation to EKS, and even then my interaction with it was pretty limited so I have no idea how it fares as a standalone operating system. My one big experience with it was the recent bug which (as I recall) attempted to harden the system by marking memory pages as no-execute, which caused virtual runtime languages like Java to basically break entirely when running on a node using this version of Bottlerocket. It was fixed pretty quickly, but it did feel like a weird thing to slip through...
Volundr 2y ago
I haven't used it personally, but https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne... looks amazing as a way to setup and manage kubernetes on Hetzner. At the moment I'm on Oracle free tier, but I keep thinking about switching to it to get off... Well Oracle.
- not_elodin 2y ago
  I've used this to set up a cluster to host a dogfooded journalling site. In one evening I had a cluster working. It works pretty well. I had one small problem when the auto-update wouldn't run on arm nodes which stopped the single node I had running at that point (with the control plane taint blocking the update pod running on them).
- mkreis 2y ago
  I'm running two clusters on it, on for production and one for dev. Works pretty good. With a schedule to reboot machines every sunday for automatic security updates (SuSE Micro OS). Also expanded machines for increased workloads. You have to make sure to inspect every change terraform wants to do, but then you're pretty save. The only downside is that every node needs a public IP, even though they are behind a firewall. But that is being worked on.
- maestrae 2y ago
  i recently read an article about running k8s on the oracle free tier and was looking to try it. i'm curious, are there any specific pain points that are making you think of switching?
  - Volundr 2y ago
    Nope, just Oracle being a corp with a nasty reputation. Honesty it was easy to set up and has been super stable, and if you go ARM the amount of resources you get for free is crazy. I actually do recommend it for personal projects on the like. I'd just be hesitant about building a business based on any Oracle offering.
chipdart 2y ago
I loved the article. Insightful, and packed with real world applications. What a gem. I have a side-question pertaining to cost-cutting with Kubernetes. I've been musing over the idea of setting up Kubernetes clusters similar to these ones but mixing on-premises nodes with nodes from the cloud provider. The setup would be something like: - vCPUs for bursty workloads, - bare metal nodes for the performance-oriented workloads required as base-loads, - on-premises nodes for spiky performance-oriented workloads, and dirt-cheap on-demand scaling. What I believe will be the primary unknown is egress costs. Has anyone ever toyed around with the idea?
- kgdkhxkzh 2y ago
  [flagged]
- oblio 2y ago
  I'm a bit sad the aggressive comment by the new account was deleted :-( The comment was making fun of the wishful thinking and the realities of networking. It was a funny comment :-(
  - rad_gruchalski 2y ago
    It wasn’t funny. I can still see it. The answer was vpn. If you want to go fancy you can do istio with vms.
    - ffsm8 2y ago
      And if you wanna be lazy, there is a tailscale integration to run the cluster communication over it. https://tailscale.com/kb/1236/kubernetes-operator https://tailscale.com/kb/1236/kubernetes-operator They've even improved it, so you can now actually resolve the services etc via the tailnet dns https://tailscale.com/learn/managing-access-to-kubernetes-with-tailscale https://tailscale.com/learn/managing-access-to-kubernetes-wi... I haven't tried that second part though, only read about it.
      - 2y ago
slillibri 2y ago
When I worked in web hosting (more than 10 years ago), we would constantly be blackholeing Hetzner IPs due to bad behavior. Same with every other budget/cheap vm provider. For us, it had nothing to do with geo databases, just behavior. You get what you pay for, and all that.
- oblio 2y ago
  They could put the backend on Hetzner, if it makes sense (for example queues or batch processors).
  - deleted 2y ago
    [deleted]
- SoftTalker 2y ago
  Yep I had the same problem years ago when I tried to use Mailgun's free tier. Not picking on them, I loved the features of their product but the free tier IPs had a horrble reputation and mail just would not get accepted especially by hotmail or yahoo. Any free hosting service will be overwhelmed by spammers and fraudsters. Cheap services the same but less so, and the more expensive they are the less they will be used for scams and spams.
  - thwarted 2y ago
    Tragedy of the Commons Ruins Everything Around Me.
- Keyframe 2y ago
  depending on the prices, maybe a valid strategy would be to have servers at hetzner and then tunnel ingress/egress somewhere more prominent. Maybe adding the network traffic to the calculation still makes financial sense?
  - srockets 2y ago
    At 0.02$/GB, it rarely does.
hipadev23 2y ago
Be careful with Hetzner, they null routed my game server on launch day due to false positives from their abuse system, and then took 3 days for their support team to re-enable traffic. By that point I had already moved to a different provider of course.
- teitoklien 2y ago
  where did you move, asking to keep a list of options for my game servers, i’m using ovh game servers atm
  - hipadev23 2y ago
    I went back to AWS. Expensive but reliable and support I can get ahold of. I’d still like to explore OVH someday though.
    - teitoklien 2y ago
      Nothing beats aws tbh, the level of extra detail aws adds, like emailing and alerting a gazillion times before making any changes to underlying hardware, even if non disruptive. Robust <24 hour support from detailed, experienced and technical support staff, a very visible customer obsession laced experience all-around. Ovh has issues with randomly taking down vps/baremetal instances at random with their support staff having no clue/late non-real time data on their instance state, they lost a ton of customer data in their huge datacenter fire 2 yrs ago, didnt even replicate the backups across multiple datacentres like they were supposed to, got sued a ton too. I use OVH because the cost reduction supremely adds up for my workloads (remote video editing/ custom rendering farm at scale with a lot more cheaper OVH s3 suitable for my temporary but too many asset workload with high egress requirements) but otherwise I miss AWS and get now, just how much superior their support and attention to detail is.
      - p_l 2y ago
        I wonder how out of norm my AWS experience is based on the comments here... Said experience being that the very highly paid support team ghosted us when we asked questions guided by their docs (AWS Workspace stuff), probably after finding that there are no answers and that we went with service based on promise of feature that apparently didn't exist
Scotrix 2y ago
Very nicely written article. I’m also running a k8s cluster but on bare metal and qemu-kvms for the base load. Wonder why you would chose VMs instead of bare metal if you looking for cost optimisation (additional overhead maybe?), could you share more about this or did I miss it?
- BillFranklin 2y ago
  Thank you! The cloud servers are sufficiently cheap for us that we could afford the extra flexibility we get from them. Hetzner can move around VMs without us noticing but in contrast they are rebooting a number of metal machines for maintenance now and for the last little while, which would have been disruptive especially during the migration. I might have another look next year at metal but I’m happy with the cloud VMs currently.
  - karussell 2y ago
    Note, they usually do not reboot or touch your servers. But yes, the current maintenance of their metal routers (rare, like once every 2 years) requires you to juggle a bit with different machines in different datacenters.
kakoni 2y ago
Anybody running k3s/k8s on Hetzner using cax servers? How's that working?
- lucasrattz 2y ago
  We are running them at https://syself.com https://syself.com for six months now, after adding support for it on our platform. So far so good.
MuffinFlavored 2y ago
https://github.com/puppetlabs/puppetlabs-kubernetes https://github.com/puppetlabs/puppetlabs-kubernetes What do the fine people of HN think about the size/scope/amount of technology of this repo? It is referenced in the article here: https://github.com/puppetlabs/puppetlabs-kubernetes/compare/main...bilbof:puppetlabs-kubernetes:main#diff-50ae7fb3724b662b58dbc1c71663cb16a484ab36aecd5a11317fb14465f847fa https://github.com/puppetlabs/puppetlabs-kubernetes/compare/...
- mkesper 2y ago
  Honestly I was surprised to hear about puppet at all. Thought that was dead and buried, like chef.
  - bigfatkitten 2y ago
    Both are still extremely widely used.
- KaiserPro 2y ago
  Puppet's original design was that it was meant to be agent based on the things it was meant to configure. It was never very good at bringing up stuff before the agent could be connected. The general flow was Imager->pre-configured puppet agent->connect to controller->apply changes to make it perform as x originally it never really had the capacity to kick off the imaging/instantiation. THis meant that it scaled better (shared state is better handled than ansible) However ansible shined because although it was a bastard to get running on more than a couple of hundred hosts in any speed, you could tell it to spin up 100x EC2(or equivalent) machines and then transform them into which every role that was needed. In puppet that was impossible to do in one go. I assume thats changed, but I don't miss puppet.
  - bigfatkitten 2y ago
    It's not really that much of an embuggerance. Kickstart or cloud-init to get the OS up and Puppet agent installed, then let Puppet do the rest.
ArtTimeInvestor 2y ago
Can anybody speak to the pros and cons of Hetzner vs OVH? There ain't many large European cloud companies, and I would like to understand how they differentiate. Ionos is another European one. Currently, it looks like their cloud business is stagnating, though.
- thenaturalist 2y ago
  I'd say stay clear of Ionos. Bonkers first experience in the last two weeks. Graphical "Data center designer", no ability to open multiple tabs, instead always rerouting to the main landing page. Attached 3 IGWs to a box, all public IPs, GUI shows "no active firewall rules". IGW 1: 100% packet loss over 1 minute. IGW 2: 85% packet loss over 1 minute. IGW3: 95% packet loss over 1 minute. Turns out "no active Firewall rules" just wasn't the case and explicit whitelisting is absolutely required. But wait, there's more! Created a hosted PostgreSQL instance, assigned a private subnet for creation. SSH into my server, ping the URL of the created Postgres instance: The DB's IP is outside the CIDR range of the assigned subnet and unreachable. What? Deleted the instance, created another one, exact same settings. Worked this time around. Support quality also varies extremely. Out of 3 encounters, I had a competent person once. Other two straight out said they have no idea what's going on.
  - ArtTimeInvestor 2y ago
    It is not possible to configure the setup without the graphical interface? Are there cloud providers you prefer?
    - mkesper 2y ago
      You could use their cloud API https://api.ionos.com/docs/cloud/v6/ https://api.ionos.com/docs/cloud/v6/ or e.g. terraform provider: https://docs.ionos.com/reference/config-management-tools/config-management-tools https://docs.ionos.com/reference/config-management-tools/con... I don't have any practical experience with this provider, though.
- Aachen 2y ago
usrme 2y ago
This is probably out of left field, but what is the benefit of having a naming scheme for nodes without any delimiters? Reading at a glance and not knowing the region name convention of a given provider (i.e. Hetzner), I'm at a loss to quickly decipher the "<region><zone><environment><role><number>" to "euc1pmgr1". I feel like I'm missing something because having delimiters would make all sorts of automated parsing much easier.
- BillFranklin 2y ago
  Quicker to type and scan! Though I admit this is preference, delimiters would work fine too. Parsing works the same but is based on a simple regex rather than splitting on a hyphen. euc=eu central; 1=zone/dc; p=production; wkr=worker; 1=node id
  - usrme 2y ago
    Thanks for getting back to me! Now that you've written it out, it's plainly obvious, but for me the readability and flexibility of delimiters beats the speed of typing and scanning. Many a times I've been grateful that I added delimiters because then I was no longer be hamstrung by any potential changes to the length of any particular segment within the name.
    - adastra22 2y ago
      You can more easily double-click-select the full hostname when there are no delimiters.
  - stackskipton 2y ago
    Yea, not putting in delimiter and then us having to change our format has bitten me so many times. Delimiter or bust.
- o11c 2y ago
  You can treat the numeric parts as self-delimiting ... that leaves only the assumption that "environment" is a single letter.
devops000 2y ago
Did you try Cloud66 for deploy?
aravindputrevu 2y ago
Do you know that they are cutting their free tier bandwidth? Did not read too much into it, but heard a few friends were worried about. End of they day, they are a business!
- lucasrattz 2y ago
  It seems to be only for the US-based servers. Sounds like they talked with a pricing consultant :p
tutfbhuf 2y ago
I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate. From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs. On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage, MariaDB operators, Cilium for networking, and ArgoCD for deploying Helm charts. We had to handle Kubernetes cluster updates ourselves, which included facing a complete cluster failure at one point. We also encountered various bugs in both Kubernetes and Ceph, many of which were documented in GitHub issues and Ceph trackers. The list of tasks to manage and monitor was endless. Depending on the number of workloads and the overall complexity of the environment, maintaining such a setup can quickly become a full-time job for a DevOps team. In contrast, using AWS or other major cloud providers allows for a more hands-off setup. With managed services, maintenance often requires significantly less effort, reducing the operational burden on your team. In essence, with AWS, your DevOps workload is reduced by a significant factor, while on Hetzner, your cloud bill is significantly lower. Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings.
- spwa4 2y ago
  > Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings. Sure, but the TLDR is going to be that if you employ n or more sysadmins, the cost savings will dominate. With 2 < n < 7. So for a given company size, Hetzner will start being cheaper at some point, and it will become more extreme the bigger you go. Second if you have a "big" cost, whatever it is, bandwidth, disk space (essentially anything but compute), cost savings will dominate faster.
  - stackskipton 2y ago
    Not always. Employing Sysadmins doesn't mean Hetzner is cheaper because those "Sysadmin/Ops type people" are being hired to managed the Kubernetes cluster. And Ops type people who truly know Kubernetes are not cheap. Sure, you can get away with legoing some K3S stuff together for a while but one major outage later, and that cost saving might have entirely disappeared.
jonas21 2y ago
This is an interesting writeup, but I feel like it's missing a description of the cluster and the workload that's running on it. How many nodes are there, how much traffic does it receive, what are the uptime and latency requirements? And what's the absolute cost savings? Saving 75% of $100K/mo is very different from saving 75% of $100/mo.
- jpgvm 2y ago
  In my experience noone bothers unless they are using GPUs or they are already at 100k/mo. I do think 100k/mo is the tipping point actually, that is $1.2M/yr. It costs around $400k/yr in engineering salaries to reasonably support a sophisticated bare metal deployment (though such people can generally do that AND provide a lot of value elsewhere in the business, so really it's actual cost is lower than this) and about $100k/yr in DC commitments, HW amortisation, and BW roughly. So you save around $700k a year which is great but the benefit becomes much greater when your equiv cloud spend is even bigger than that.
  - wrd83 2y ago
    If you want to do HA kubernetes, you need oncalls and at least 10 engineers to get a stable rotation. If you do that in Europe you have to pay them during standby hours. 400k/year seems very low to me.
    - jpgvm 2y ago
      You really don't need all 10 people on-call to know k8s to that level. They just need to know enough as to when to wake someone else up. Everywhere I have worked where we have run clusters in the 100s to 1000s of nodes we have rarely had a team larger than 4-5 of true k8s folks and even then it's been a split between folks that are very hardware provisioning/network/etc focused and more higher level k8s folk which also take on a large portion of CI/CD work also. At smaller scale (in the $1M/yr ballpark) I have done all the k8s bare metal ops myself along with all CI/CD and been responsible for a ton of the backend programming too. This is feasible because with distros like Talos etc it doesn't take a lot of manpower once it's setup and upgrades aren't too painful at small scale if you aren't running stateful services. So tbh no, you just need ideally 2 folks at around ~200k/yr each that are competent and have done it before. The rest of the folks on the on-call rotation are just the rest of your engineers (and if you are at $1m/yr cloud spend you have more than 10 of those).
sureglymop 2y ago
I went hetzner baremetal, set up a proxmox cluster over it and then have kubernetes on top. Gives me a lot of flexibility I find.
Iwan-Zotow 2y ago
this is good well, running on bare metal would be even better
- lucasrattz 2y ago
  You could take a look at https://syself.com https://syself.com - 100% support for bare metal (I'm an employee).
  - Iwan-Zotow 2y ago
    thanks, will look
no_carrier 2y ago
> While DigitalOcean, like other providers, offers a free managed control plane, there is typically a 100% markup on the nodes that belong to these managed clusters. I don't think this is true. With Digital Ocean, the worker nodes are the same cost as regular droplets, there's no additional costs involved. This makes Digital Ocean's offering very attractive - free control plane you don't have to worry about, free upgrades, and some extra integrations to things like the load balancer, storage, etc. I can't think of a reason to not go with that over self-managed.
- czhu12 2y ago
  The actual nodes are still way more expensive on digital ocean than they are in Hetzner. That’s probably the main reason. 8GB RAM, shared cpu on hetzner is ~$10 Equivalent on digital ocean is $48
- lucasrattz 2y ago
  Besides what czhu12 mentioned, DOKS charging extra for HA control planes make me feel as if the platform is not production-grade. If you want a managed experience on Hetzner, you could take a look at https://syself.com https://syself.com Disclaimer: I'm an employee there
  - theogre 2y ago
    [dead]
s3rius 2y ago
That's a really good article. Actually, recently we were migrating as well and we were using dedicated nodes in our setup. In order to integrate a load-balancer provided by hetzner with our k8s on dedicated servers we had to implement a super thin operator that does it: https://github.com/Intreecom/robotlb https://github.com/Intreecom/robotlb If anyone will be inspired by this article and would want to do the same, feel free to use this project.
czhu12 2y ago
Funnily enough, we made the exact same transition from heroku to DigitalOceans managed Kubernetes service, and saved about 75%. Presumably this means that had we moved from heroku to hetzner, it would have been 93% savings! The costs of cloud hosting are totally out of control, would love to see more efforts that lets developers move down the stack. I’ve been humbly working on https://canine.sh https://canine.sh which basically provides a Heroku like interface to any K8 cluster
esher 2y ago
As far as I see, no one is mentioning sustainability AKA environmental impact or 'green hosting' here. Don't you care about that? I believe that Hetzner data centers in Europe (Germany, Finland) are powered by green energy, but not the locations in US.
- huijzer 2y ago
  Data centers used 460 TWh, or about 2% of total worldwide electricity use, according to IEA in 2022. In comparison, 30% of total energy (energy! Not electricity) goes to transport! As another point of comparison, transport in Sweden in 2022 used 137 TWh [1]. So the same order of magnitude as total datacenter energy use. And datacenters are powered by electricity which increases the chance that it comes from renewable energy. Conversely, the chance that diesel comes from a renewable source is zero. So can we please stop talking about data center energy use? It’s a narrative that the media is currently pushing but as so many things it makes no sense. It’s not the thing we should be focusing on if we want to decrease fossil fuel use. [1]: https://www.energimyndigheten.se/en/energysystem/energy-consumption/ https://www.energimyndigheten.se/en/energysystem/energy-cons...
  - esher 2y ago
    Thanks for sharing. I care about it. I run a small hosting company. Sure, there are many low hanging fruits for fighting CO2 emissions that should be tackled first. I am also hopeful that energy from directly available renewables will be the most economic choice for building data centers anyhow, so that this is not a matter of believes any more. But on the other side, to bring down CO2 levels, fast change everywhere is required. As far as I see data center energy consumption continues to grow, specifically with AI. If I am not mistaken, data centers produce more CO2 than aviation. And sure, most 'green hosting' is probably 'green washing', yet I would still support and link initiatives such as: https://www.thegreenwebfoundation.org/ https://www.thegreenwebfoundation.org/
  - davedx 2y ago
    2% of total worldwide electricity use in 2022 is a shit load of electricity and emissions. Your argument is the same as those who argue "our country shouldn't care about emissions when China is the biggest emitter". If you dive into a detailed breakdown of emissions you'll find that it's a complex hierarchy of categories. You can't just fix "all of transport" or treat it like a "low hanging fruit", just look at how much time it's taken for EV penetration to be in any way significant; look at how much of transport emissions are from aviation or shipping or other components. Any energy use that's measurable in whole percentage points of global emissions needs addressing. That includes data centers.
surrTurr 2y ago
> Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner's volumes we were seeing >50ms of IOWAIT with very low IOPS. See https://github.com/rook/rook/issues/14999 https://github.com/rook/rook/issues/14999 for benchmarks. I set up rook ceph on a talos k8s cluster (with vm volumes) and experienced similar low performance; however, I always thought that was because of the 1Gi vSwitch (i.e. networking problem)?! The SSD volumes were quite fast.
- merpkz 2y ago
  In my limited experience with rook-ceph it is strictly bare metal technology to deploy. On virtualization it will basically replicate your data to VM disks which usually are already replicated, so quite a bit of replication amplification will happen and tank your performance.
- tehlike 2y ago
  SSD volumes are physically on the same node, and afaik not redundant. The cloud vms are ceph clusters behind the scenes, and writes need to commit for 3+ machines. It's both network latency and inherent process latency Additionally, hetzner has an IOPS limit of 5000 and write limit of some amount that does not scale with the size of database. 50G has the same limits as 5TB. For this reason, people are sometimes using different table spaces in postgres for example. Ceph puts another burden on top of already-ceph-based cloud volumes, btw, so don't do that.
  - p_l 2y ago
    RAID10 on local SSDs is pretty performant option, but yeah, it's per node.
mythz 2y ago
Been a happy Hetzner customer for over a decade, previously using their dedicated servers in their German DC's before migrating to their Cloud US VMs for better latency with the US. Slightly disappointed with their recent cuts of their generous 20TB free traffic down to 3TB (€1.19 per additional TB), but they still look to be a lot better value than all other US cloud providers we've evaluated. Whilst I wouldn't run Kubernetes by choice, we've had success moving our custom SSH / Docker compose deployments over to use GitHub Actions with kamal-deploy.org, easy to setup and nice UX tools for monitoring remote deployed apps [1] [1] https://servicestack.net/posts/kamal-deployments https://servicestack.net/posts/kamal-deployments
- Voultapher 2y ago
  Seems to be a US thing, maybe their peering partners are forcing them to raise prices, the German DC still stells the 20TB bandwidth https://www.hetzner.com/cloud/ https://www.hetzner.com/cloud/, but US is an order of magnitude less for the same price :/
  - inemesitaffia 2y ago
    I don't see how traffic in Ashburn is more expensive than Frankfurt and Amsterdam. It's the sort of place where people say Transit is cheaper than paid peering. (For eyeball networks at least). I think carrying traffic from Europe for some images and videos might make sense financially. But there's always bulk CDN's
    - kuschku 2y ago
      > I don't see how traffic in Ashburn is more expensive than Frankfurt and Amsterdam. The vast majority of Hetzner's traffic in europe (and tbh, anyone's traffic) is free peering. Telekom is the one major exception.
bittermandel 2y ago
We're very happy to use Hetzner for our bare-metal staging environments to validate functionality, but I still feel reluctant to put our production there. Disks don't quite work as intended at all times and our vSwitch setup has gotten reset more than once. All of this makes sense considering the extremely low price.
adamcharnock 2y ago
We've [1] been using Hetzner's dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we typically see request times half. And because the hardware is cheaper we can provide dedicated DevOps engineering time to each client. There are some caveats though: 1) A staging cluster for testing updates is really a must. YOLO-ing prod updates on a Sunday is no one's idea of fun. 2) Application level replication is king, followed by block-level replication (we use OpenEBS/Mayastor). After going through all the Postgres operators we found StackGres to (currently) be the best. 3) The Ansible playbooks are your assets. Once you have them down and well-commented for a given service then re-deploying that service in other cases (or again in the future) becomes straightforward. 4) If you can I'd recommend a dedicated 10G network to connect your servers. 1G just isn't quite enough when it comes to the combined load of prod traffic, plus image pulls, plus inter-service traffic. This also gives a 10x latency improvement over AWS intra-az. 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). 6) MinIO clusters (via the operator) are not that tricky to operate as long as you follow the well trodden path. This provides you with local high-bandwidth, low-latency object storage. 7) The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time. 8) You can still push ancillary/annoying tasks off onto cloud providers (personally I'm a fan of CloudFlare for HTTP load balancing). [1]: https://lithus.eu https://lithus.eu
- sureIy 2y ago
  > I'd put it at 2-4 months of undistracted skilled engineering time. How much is that worth to your company/customer vs a higher monthly bill for the next 5 years? As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all, unless I expect a 10k/month AWS bill. xkcd comes to mind: https://xkcd.com/1319/ https://xkcd.com/1319/
  - adamcharnock 2y ago
    > As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all. Well I do rather agree, but as a consultancy I'm biased. But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers). Say one wants a 3 year payback, that is $2,200/month savings you need. Which seems highly achievable given some of the cloud spends I've seen, and that I think an 80-90% reduction in cloud spend is a good ballpark. The appeal of a consultancy is that we'll remove the up-front investment, provide the skills, de-risk the whole endeavour, even put engineers within your team, but you'll _only_ save 50%. The latter option is much more appealing in terms of hiring, risk, and cash-flow. But if your company has the skills, the cash, and the risk tolerance then maybe the former approach is best. EDIT: I actually think the(/our) consultancy option is a really good idea for startups. Their infrastructure ends up being slightly over-built to start with, but very quickly they end up saving a lot of money, and they also get DevOps staffing without having to hire for it. Moreover, the DevOps resource available to them scales with their compute needs. (also we offer 2x the amount of DevOps days for startups for the first year to help them get up and running).
mnming 2y ago
I feel lots of the work described in the article can be automated by kops, probably in a much better way, especially when it comes to day 2 operations. I wonder what is the motivation behind manually spinning up a cluster instead of going with more established tooling?
Hetzner_OL 2y ago
Hi Bill, Wow! Thanks for the amazing write-up and for sharing it on your blog and here! I am so happy that we've helped you save so much money and that you're happy with our support team! It's a great way to start off the week! --Katie
Neil44 2y ago
When I first started hosting servers/services for customers I was using EC2 and Rackspace, then I discovered Linode and was happy it was so much cheaper with apparently no downside. After the first couple of interactions with support I started to relax. Then I discovered OVH, same story. I haven't needed the support yet though.
wvh 2y ago
I work for a consultancy company that helps companies building and securing infrastructure. We have a lot of customers running Kubernetes at low-cost providers (like Hetzner), more local middle-tier and top-three (AWS, GCP, Azure). We also have some governmental, financial and medical companies that can not or will not run in public clouds, so they usually host on-prem. If Hetzner has an issue or glitch once a month, the middle-tier providers have one every 2-3 months, and a place like AWS maybe every 5-6 months. However, prices also follow that observation, so you have to carefully consider on a case-by-case basis whether adding some extra machines and backup and failure scenarios is a better deal. The major benefit by using basic hosting services is that their pricing is a lot more predictable; you pay for machines and scale as you go. Once you get hooked into all the extra services a provider like AWS provides, you might get some unexpectedly high bills and moving away might be a lot harder. For smaller companies, don't make short-sighted decisions that threaten your ability to survive long-term by choosing the easy solution or "free credits" scheme early on. There is no right answer here, just trade-offs.
jillesvangurp 2y ago
The key take home point here is not how amazingly cheap Hetzner is, which it is. But how much of an extortion game Google, Amazon, MS, etc. are playing with their cloud services. These are trillion dollar companies because they are raking in cash with extreme margins. Yes, there is some added value in the level of convenience provided. But maybe with a bit more competition, pricing could be more competitive. A lot more competitive.
acac10 2y ago
// Taking another slant at the discussion: Why kubernetes? Thank you for sharing your experience. I also have my 3 personal servers with Hetzner, plus a couple VM instances in Scaleways (French outfit). Disclaimer: I’m a Googler, was SRE for ~10 years for GMail, identity, social, apps (gsuites nowadays) and more, managed hundreds of jobs in Borg, one of the 3 founders of the current dev+devops internal platform (and I focused on the releases,prod,capacity side of the platform), dabbled in K8s on my personal time. My opinions, not Google’s. So, my question is: given the significant complexity that K8s brings (I don’t think anyone disputes this) why are people using it outside medium-large environments? There are simpler and yet flexible & effective job schedulers that are way easier to manage. Nomad is an example. Unless you have a LOT of machines to manage, with many jobs (I’d say +250) to manage, K8s complexity, brittleness and overhead are not justifiable, IMO. The emergence of tools like Terraform and the many other management layers in top of K8s that try to make it easier but just introduce more complexity and their own abstractions are in itself a sign of that inherent complexity. I would say that only a few companies in the world need that level of complexity. And then they will need it, for sure. But, for most is like buying a Formula 1 to commute in a city. One other aspect that I also noticed is that technical teams tend to carry on the mess they had in their previous “legacy” environment and just replicate in K8s, instead of trying to do an architectural design of the whole system needs. And K8s model enables that kind of mess: a “bucket of things”. Those two things combined, mean that nowadays every company has soaring cloud costs, are running things they know nothing about but are afraid to touch in case of breaking something. And an outage is more career harming than a high bill that Finance will deal with it later, so why risk it, right? A whole new IT area has been coined now to deal with this: FinOps :facepalm: I’m just puzzled by the whole situation, tbh.
- KaiserPro 2y ago
  I too used to run a large clustered environment (VFX) and now work at a FAANG which has a "borg-like" scheduler. K8s has a whole kit of parts which sound really grand when you are starting out on a new platform, but quickly become a pain when you actually start to implement it. I think thats the biggest problem, is by the time you've realised that actualy you don't need k8s, you've invested so much time into learning the sodding thing, its difficult to back out. The other seductive thing is helm provides "AWS-like" features (ie fancy load balancing rules) that are hard to figure out unless you've dabbled with the underlying tech before (varnish/nginx/etc are daunting, so is storage and networking) this tends to lead to utterly fucking stupid networking systems because unless you know better, that looks normal.
james_sulivan 2y ago
For those considering Hetzner, there is also Contabo which is another German hosting company that is also good, at least in my experience
0xbadcafebee 2y ago
I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time "re-fixing" things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? Some things are not worth DIYing; pay a mechanic or lease your car, especially if you depend on it for your livelihood. Even something as simple as an oil change, really isn't worth doing yourself. First you buy the tools (oil drip pan, filter wrench, funnel, creeper). Then you set aside the time to use them, find your dingy work clothes. You go to the store and buy new oil and a filter. You go home and change the oil. Then that day or another day you go to a store that will take your used oil. Versus 20 minutes at an auto mechanic, for about $15 more than the cost of the oil and filter. Kubernetes is an entire car (and a complex one). It's really not worth doing the maintenance yourself, I promise you. Unless you're just doing it for fun.
- p_l 2y ago
  Unless you find out that calling AA for everything and going to big name mechanics garage every time costs you significant portion of your budget. A lot of it is finding balance between what to do yourself, what to outsource, and it's not as easy or clean as some people here like to claim.
  - infecto 2y ago
    +1 to this. I agree with the takeaway that paying for managed Kubernetes is the way to go 9 times out of 10 but I think in life its quite powerful to attempt to do things yourself sometimes. No, you should not go out and buy thousands of dollars worth of tools and do everything yourself but its pretty powerful to be able to do simple things in your life...change oil, replace light switches/sockets. The reason I personally find it powerful is that it helps me quantify what is worth outsourcing and what is not and it also builds mental models for how things work which is important when outsourcing work.
- theappsecguy 2y ago
  I don’t know. My shop wanted 1800 to change my brakes. I bought the parts for 300 and got it done in a day (first time). Seems like a pretty good payback and good skill to have. My neighbour has a car lift which certainly helped.
awinter-py 2y ago
cut my kube bill 100% on GKE by switching from regional to zonal cluster bc the first zonal cluster is free
deleted 2y ago
[deleted]
lucasrattz 2y ago
We at Syself.com also have great experiences with Kubernetes on Hetzner. We built a platform on top of Cluster API and brought a managed Kubernetes experience to Hetzner. Now we have self-healing, automated updates and 100% reproductibility, with full bare metal support. > Hetzner volumes are, in my experience, too slow for a production database. That's true, though. To solve that we developed a way to persist the local storage of bare metal servers across reprovisionings. This way it's both faster and cheaper. Now we are adding an automated database deployment layer on top of it.

375 points by BillFranklin 2y ago

deleted 2y ago

[deleted]

> Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner's volumes we were seeing >50ms of IOWAIT with very low IOPS. There is a surprisingly easy way to address this issue: use (ridiculously cheap) Hetzner metal machines as nodes. The ones with nvme storage offer excellent performance for dbs and often have generous amounts of RAM. I'd go as far as to say you'd be better off to invest in two or more beefy bare metal machines for a master-replica(s) setup rather than run the db on k8s. If you don't want to be bothered with the setup, you can use one of many modern packages such as Pigsty: https://pigsty.cc/ https://pigsty.cc/ (not affiliated but a huge fan).

BillFranklin 2y ago
Thanks, hadn’t heard of pigsty. As you say, I had to use nvme ssds for the dbs, the performance is pretty good so I didn’t look to get metal nodes.
threeseed 2y ago
There are plenty of options for running a database on Kubernetes whilst using local NVMe storage. There are just pinning the database pods to specific nodes and using a LocalPathProvisioner or distributed solutions like JuiceFS, OpenEBS etc.
gourneau 2y ago
Thanks for the Pigsty link. I have been a big fan of running Postgres on metal machines.
lucasrattz 2y ago
I've had great experiences with using the bare metal server's local storage. This is the guide I wrote for our customers: https://syself.com/docs/hetzner/apalla/how-to-guides/storage/local-storage-in-baremetal https://syself.com/docs/hetzner/apalla/how-to-guides/storage...

cjr 2y ago

What about cluster autoscaling?

BillFranklin 2y ago
I didn’t touch on that in the article, but essentially it’s a one line change to add a worker node (or nodes) to the cluster, then it’s automatically enrolled. We don’t have such bursty requirements fortunately so I have not needed to automate this.
preisschild 2y ago
Works rather well. I use CAPI + Cluster-Autoscaler + Talos and new nodes are provisioned and ready within 2-3 minutes.

postepowanieadm 2y ago

Lovely website.

segmondy 2y ago

Great write up Bill!

aliasxneo 2y ago

I’m planning on doing something similar but want to use Talos with bare metal machines. I suspect to see similar price reductions from our current EKS bill.

threeseed 2y ago
Depending on your cluster size I highly recommend Omni: https://omni.siderolabs.com https://omni.siderolabs.com It took minutes to setup a cluster and I love having a UI to see what is happening. I wish there were more products like this as I suspect there will be a trend towards more self-managed Kubernetes clusters given how expensive the cloud is becoming.
MathiasPius 2y ago
I set up a Talos bare metal cluster about a year ago, and documented the whole process on my website. Feel free to reach out if you have any questions!
- cedws 2y ago
  Any thoughts/feelings about Talos vs Bottlerocket?
  - p_l 2y ago
    Did bottlerocket get any closer to stable and usable outside AWS walled garden? Last time I tried was admittedly in 2022, but in testing which distro to go with bottlerocket lost because we couldn't setup local builds...
  - MathiasPius 2y ago
    I've only used Bottlerocket in relation to EKS, and even then my interaction with it was pretty limited so I have no idea how it fares as a standalone operating system. My one big experience with it was the recent bug which (as I recall) attempted to harden the system by marking memory pages as no-execute, which caused virtual runtime languages like Java to basically break entirely when running on a node using this version of Bottlerocket. It was fixed pretty quickly, but it did feel like a weird thing to slip through...

Volundr 2y ago

I haven't used it personally, but https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner https://github.com/kube-hetzner/terraform-hcloud-kube-hetzne... looks amazing as a way to setup and manage kubernetes on Hetzner. At the moment I'm on Oracle free tier, but I keep thinking about switching to it to get off... Well Oracle.

not_elodin 2y ago
I've used this to set up a cluster to host a dogfooded journalling site. In one evening I had a cluster working. It works pretty well. I had one small problem when the auto-update wouldn't run on arm nodes which stopped the single node I had running at that point (with the control plane taint blocking the update pod running on them).
mkreis 2y ago
I'm running two clusters on it, on for production and one for dev. Works pretty good. With a schedule to reboot machines every sunday for automatic security updates (SuSE Micro OS). Also expanded machines for increased workloads. You have to make sure to inspect every change terraform wants to do, but then you're pretty save. The only downside is that every node needs a public IP, even though they are behind a firewall. But that is being worked on.
maestrae 2y ago
i recently read an article about running k8s on the oracle free tier and was looking to try it. i'm curious, are there any specific pain points that are making you think of switching?
- Volundr 2y ago
  Nope, just Oracle being a corp with a nasty reputation. Honesty it was easy to set up and has been super stable, and if you go ARM the amount of resources you get for free is crazy. I actually do recommend it for personal projects on the like. I'd just be hesitant about building a business based on any Oracle offering.

chipdart 2y ago

I loved the article. Insightful, and packed with real world applications. What a gem. I have a side-question pertaining to cost-cutting with Kubernetes. I've been musing over the idea of setting up Kubernetes clusters similar to these ones but mixing on-premises nodes with nodes from the cloud provider. The setup would be something like: - vCPUs for bursty workloads, - bare metal nodes for the performance-oriented workloads required as base-loads, - on-premises nodes for spiky performance-oriented workloads, and dirt-cheap on-demand scaling. What I believe will be the primary unknown is egress costs. Has anyone ever toyed around with the idea?

kgdkhxkzh 2y ago
[flagged]
oblio 2y ago
I'm a bit sad the aggressive comment by the new account was deleted :-( The comment was making fun of the wishful thinking and the realities of networking. It was a funny comment :-(
- rad_gruchalski 2y ago
  It wasn’t funny. I can still see it. The answer was vpn. If you want to go fancy you can do istio with vms.
  - ffsm8 2y ago
    And if you wanna be lazy, there is a tailscale integration to run the cluster communication over it. https://tailscale.com/kb/1236/kubernetes-operator https://tailscale.com/kb/1236/kubernetes-operator They've even improved it, so you can now actually resolve the services etc via the tailnet dns https://tailscale.com/learn/managing-access-to-kubernetes-with-tailscale https://tailscale.com/learn/managing-access-to-kubernetes-wi... I haven't tried that second part though, only read about it.
    - 2y ago

slillibri 2y ago

When I worked in web hosting (more than 10 years ago), we would constantly be blackholeing Hetzner IPs due to bad behavior. Same with every other budget/cheap vm provider. For us, it had nothing to do with geo databases, just behavior. You get what you pay for, and all that.

oblio 2y ago
They could put the backend on Hetzner, if it makes sense (for example queues or batch processors).
- deleted 2y ago
  [deleted]
SoftTalker 2y ago
Yep I had the same problem years ago when I tried to use Mailgun's free tier. Not picking on them, I loved the features of their product but the free tier IPs had a horrble reputation and mail just would not get accepted especially by hotmail or yahoo. Any free hosting service will be overwhelmed by spammers and fraudsters. Cheap services the same but less so, and the more expensive they are the less they will be used for scams and spams.
- thwarted 2y ago
  Tragedy of the Commons Ruins Everything Around Me.
Keyframe 2y ago
depending on the prices, maybe a valid strategy would be to have servers at hetzner and then tunnel ingress/egress somewhere more prominent. Maybe adding the network traffic to the calculation still makes financial sense?
- srockets 2y ago
  At 0.02$/GB, it rarely does.

hipadev23 2y ago

Be careful with Hetzner, they null routed my game server on launch day due to false positives from their abuse system, and then took 3 days for their support team to re-enable traffic. By that point I had already moved to a different provider of course.

teitoklien 2y ago
where did you move, asking to keep a list of options for my game servers, i’m using ovh game servers atm
- hipadev23 2y ago
  I went back to AWS. Expensive but reliable and support I can get ahold of. I’d still like to explore OVH someday though.
  - teitoklien 2y ago
    Nothing beats aws tbh, the level of extra detail aws adds, like emailing and alerting a gazillion times before making any changes to underlying hardware, even if non disruptive. Robust <24 hour support from detailed, experienced and technical support staff, a very visible customer obsession laced experience all-around. Ovh has issues with randomly taking down vps/baremetal instances at random with their support staff having no clue/late non-real time data on their instance state, they lost a ton of customer data in their huge datacenter fire 2 yrs ago, didnt even replicate the backups across multiple datacentres like they were supposed to, got sued a ton too. I use OVH because the cost reduction supremely adds up for my workloads (remote video editing/ custom rendering farm at scale with a lot more cheaper OVH s3 suitable for my temporary but too many asset workload with high egress requirements) but otherwise I miss AWS and get now, just how much superior their support and attention to detail is.
    - p_l 2y ago
      I wonder how out of norm my AWS experience is based on the comments here... Said experience being that the very highly paid support team ghosted us when we asked questions guided by their docs (AWS Workspace stuff), probably after finding that there are no answers and that we went with service based on promise of feature that apparently didn't exist

Scotrix 2y ago

Very nicely written article. I’m also running a k8s cluster but on bare metal and qemu-kvms for the base load. Wonder why you would chose VMs instead of bare metal if you looking for cost optimisation (additional overhead maybe?), could you share more about this or did I miss it?

BillFranklin 2y ago
Thank you! The cloud servers are sufficiently cheap for us that we could afford the extra flexibility we get from them. Hetzner can move around VMs without us noticing but in contrast they are rebooting a number of metal machines for maintenance now and for the last little while, which would have been disruptive especially during the migration. I might have another look next year at metal but I’m happy with the cloud VMs currently.
- karussell 2y ago
  Note, they usually do not reboot or touch your servers. But yes, the current maintenance of their metal routers (rare, like once every 2 years) requires you to juggle a bit with different machines in different datacenters.

kakoni 2y ago

Anybody running k3s/k8s on Hetzner using cax servers? How's that working?

lucasrattz 2y ago
We are running them at https://syself.com https://syself.com for six months now, after adding support for it on our platform. So far so good.

MuffinFlavored 2y ago

https://github.com/puppetlabs/puppetlabs-kubernetes https://github.com/puppetlabs/puppetlabs-kubernetes What do the fine people of HN think about the size/scope/amount of technology of this repo? It is referenced in the article here: https://github.com/puppetlabs/puppetlabs-kubernetes/compare/main...bilbof:puppetlabs-kubernetes:main#diff-50ae7fb3724b662b58dbc1c71663cb16a484ab36aecd5a11317fb14465f847fa https://github.com/puppetlabs/puppetlabs-kubernetes/compare/...

mkesper 2y ago
Honestly I was surprised to hear about puppet at all. Thought that was dead and buried, like chef.
- bigfatkitten 2y ago
  Both are still extremely widely used.
KaiserPro 2y ago
Puppet's original design was that it was meant to be agent based on the things it was meant to configure. It was never very good at bringing up stuff before the agent could be connected. The general flow was Imager->pre-configured puppet agent->connect to controller->apply changes to make it perform as x originally it never really had the capacity to kick off the imaging/instantiation. THis meant that it scaled better (shared state is better handled than ansible) However ansible shined because although it was a bastard to get running on more than a couple of hundred hosts in any speed, you could tell it to spin up 100x EC2(or equivalent) machines and then transform them into which every role that was needed. In puppet that was impossible to do in one go. I assume thats changed, but I don't miss puppet.
- bigfatkitten 2y ago
  It's not really that much of an embuggerance. Kickstart or cloud-init to get the OS up and Puppet agent installed, then let Puppet do the rest.

ArtTimeInvestor 2y ago

Can anybody speak to the pros and cons of Hetzner vs OVH? There ain't many large European cloud companies, and I would like to understand how they differentiate. Ionos is another European one. Currently, it looks like their cloud business is stagnating, though.

thenaturalist 2y ago
I'd say stay clear of Ionos. Bonkers first experience in the last two weeks. Graphical "Data center designer", no ability to open multiple tabs, instead always rerouting to the main landing page. Attached 3 IGWs to a box, all public IPs, GUI shows "no active firewall rules". IGW 1: 100% packet loss over 1 minute. IGW 2: 85% packet loss over 1 minute. IGW3: 95% packet loss over 1 minute. Turns out "no active Firewall rules" just wasn't the case and explicit whitelisting is absolutely required. But wait, there's more! Created a hosted PostgreSQL instance, assigned a private subnet for creation. SSH into my server, ping the URL of the created Postgres instance: The DB's IP is outside the CIDR range of the assigned subnet and unreachable. What? Deleted the instance, created another one, exact same settings. Worked this time around. Support quality also varies extremely. Out of 3 encounters, I had a competent person once. Other two straight out said they have no idea what's going on.
- ArtTimeInvestor 2y ago
  It is not possible to configure the setup without the graphical interface? Are there cloud providers you prefer?
  - mkesper 2y ago
    You could use their cloud API https://api.ionos.com/docs/cloud/v6/ https://api.ionos.com/docs/cloud/v6/ or e.g. terraform provider: https://docs.ionos.com/reference/config-management-tools/config-management-tools https://docs.ionos.com/reference/config-management-tools/con... I don't have any practical experience with this provider, though.
Aachen 2y ago

usrme 2y ago

This is probably out of left field, but what is the benefit of having a naming scheme for nodes without any delimiters? Reading at a glance and not knowing the region name convention of a given provider (i.e. Hetzner), I'm at a loss to quickly decipher the "<region><zone><environment><role><number>" to "euc1pmgr1". I feel like I'm missing something because having delimiters would make all sorts of automated parsing much easier.

BillFranklin 2y ago
Quicker to type and scan! Though I admit this is preference, delimiters would work fine too. Parsing works the same but is based on a simple regex rather than splitting on a hyphen. euc=eu central; 1=zone/dc; p=production; wkr=worker; 1=node id
- usrme 2y ago
  Thanks for getting back to me! Now that you've written it out, it's plainly obvious, but for me the readability and flexibility of delimiters beats the speed of typing and scanning. Many a times I've been grateful that I added delimiters because then I was no longer be hamstrung by any potential changes to the length of any particular segment within the name.
  - adastra22 2y ago
    You can more easily double-click-select the full hostname when there are no delimiters.
- stackskipton 2y ago
  Yea, not putting in delimiter and then us having to change our format has bitten me so many times. Delimiter or bust.
o11c 2y ago
You can treat the numeric parts as self-delimiting ... that leaves only the assumption that "environment" is a single letter.

devops000 2y ago

Did you try Cloud66 for deploy?

aravindputrevu 2y ago

Do you know that they are cutting their free tier bandwidth? Did not read too much into it, but heard a few friends were worried about. End of they day, they are a business!

lucasrattz 2y ago
It seems to be only for the US-based servers. Sounds like they talked with a pricing consultant :p

tutfbhuf 2y ago

I have experience running Kubernetes clusters on Hetzner dedicated servers, as well as working with a range of fully or highly managed services like Aurora, S3, and ECS Fargate. From my experience, the cloud bill on Hetzner can sometimes be as low as 20% of an equivalent AWS bill. However, this cost advantage comes with significant trade-offs. On Kubernetes with Hetzner, we managed a Ceph cluster using NVMe storage, MariaDB operators, Cilium for networking, and ArgoCD for deploying Helm charts. We had to handle Kubernetes cluster updates ourselves, which included facing a complete cluster failure at one point. We also encountered various bugs in both Kubernetes and Ceph, many of which were documented in GitHub issues and Ceph trackers. The list of tasks to manage and monitor was endless. Depending on the number of workloads and the overall complexity of the environment, maintaining such a setup can quickly become a full-time job for a DevOps team. In contrast, using AWS or other major cloud providers allows for a more hands-off setup. With managed services, maintenance often requires significantly less effort, reducing the operational burden on your team. In essence, with AWS, your DevOps workload is reduced by a significant factor, while on Hetzner, your cloud bill is significantly lower. Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings.

spwa4 2y ago
> Determining which option is more cost-effective requires a thorough TCO (Total Cost of Ownership) analysis. While Hetzner may seem cheaper upfront, the additional hours required for DevOps work can offset those savings. Sure, but the TLDR is going to be that if you employ n or more sysadmins, the cost savings will dominate. With 2 < n < 7. So for a given company size, Hetzner will start being cheaper at some point, and it will become more extreme the bigger you go. Second if you have a "big" cost, whatever it is, bandwidth, disk space (essentially anything but compute), cost savings will dominate faster.
- stackskipton 2y ago
  Not always. Employing Sysadmins doesn't mean Hetzner is cheaper because those "Sysadmin/Ops type people" are being hired to managed the Kubernetes cluster. And Ops type people who truly know Kubernetes are not cheap. Sure, you can get away with legoing some K3S stuff together for a while but one major outage later, and that cost saving might have entirely disappeared.

jonas21 2y ago

This is an interesting writeup, but I feel like it's missing a description of the cluster and the workload that's running on it. How many nodes are there, how much traffic does it receive, what are the uptime and latency requirements? And what's the absolute cost savings? Saving 75% of $100K/mo is very different from saving 75% of $100/mo.

jpgvm 2y ago
In my experience noone bothers unless they are using GPUs or they are already at 100k/mo. I do think 100k/mo is the tipping point actually, that is $1.2M/yr. It costs around $400k/yr in engineering salaries to reasonably support a sophisticated bare metal deployment (though such people can generally do that AND provide a lot of value elsewhere in the business, so really it's actual cost is lower than this) and about $100k/yr in DC commitments, HW amortisation, and BW roughly. So you save around $700k a year which is great but the benefit becomes much greater when your equiv cloud spend is even bigger than that.
- wrd83 2y ago
  If you want to do HA kubernetes, you need oncalls and at least 10 engineers to get a stable rotation. If you do that in Europe you have to pay them during standby hours. 400k/year seems very low to me.
  - jpgvm 2y ago
    You really don't need all 10 people on-call to know k8s to that level. They just need to know enough as to when to wake someone else up. Everywhere I have worked where we have run clusters in the 100s to 1000s of nodes we have rarely had a team larger than 4-5 of true k8s folks and even then it's been a split between folks that are very hardware provisioning/network/etc focused and more higher level k8s folk which also take on a large portion of CI/CD work also. At smaller scale (in the $1M/yr ballpark) I have done all the k8s bare metal ops myself along with all CI/CD and been responsible for a ton of the backend programming too. This is feasible because with distros like Talos etc it doesn't take a lot of manpower once it's setup and upgrades aren't too painful at small scale if you aren't running stateful services. So tbh no, you just need ideally 2 folks at around ~200k/yr each that are competent and have done it before. The rest of the folks on the on-call rotation are just the rest of your engineers (and if you are at $1m/yr cloud spend you have more than 10 of those).

sureglymop 2y ago

I went hetzner baremetal, set up a proxmox cluster over it and then have kubernetes on top. Gives me a lot of flexibility I find.

Iwan-Zotow 2y ago

this is good well, running on bare metal would be even better

lucasrattz 2y ago
You could take a look at https://syself.com https://syself.com - 100% support for bare metal (I'm an employee).
- Iwan-Zotow 2y ago
  thanks, will look

no_carrier 2y ago

> While DigitalOcean, like other providers, offers a free managed control plane, there is typically a 100% markup on the nodes that belong to these managed clusters. I don't think this is true. With Digital Ocean, the worker nodes are the same cost as regular droplets, there's no additional costs involved. This makes Digital Ocean's offering very attractive - free control plane you don't have to worry about, free upgrades, and some extra integrations to things like the load balancer, storage, etc. I can't think of a reason to not go with that over self-managed.

czhu12 2y ago
The actual nodes are still way more expensive on digital ocean than they are in Hetzner. That’s probably the main reason. 8GB RAM, shared cpu on hetzner is ~$10 Equivalent on digital ocean is $48
lucasrattz 2y ago
Besides what czhu12 mentioned, DOKS charging extra for HA control planes make me feel as if the platform is not production-grade. If you want a managed experience on Hetzner, you could take a look at https://syself.com https://syself.com Disclaimer: I'm an employee there
- theogre 2y ago
  [dead]

s3rius 2y ago

That's a really good article. Actually, recently we were migrating as well and we were using dedicated nodes in our setup. In order to integrate a load-balancer provided by hetzner with our k8s on dedicated servers we had to implement a super thin operator that does it: https://github.com/Intreecom/robotlb https://github.com/Intreecom/robotlb If anyone will be inspired by this article and would want to do the same, feel free to use this project.

czhu12 2y ago

Funnily enough, we made the exact same transition from heroku to DigitalOceans managed Kubernetes service, and saved about 75%. Presumably this means that had we moved from heroku to hetzner, it would have been 93% savings! The costs of cloud hosting are totally out of control, would love to see more efforts that lets developers move down the stack. I’ve been humbly working on https://canine.sh https://canine.sh which basically provides a Heroku like interface to any K8 cluster

esher 2y ago

As far as I see, no one is mentioning sustainability AKA environmental impact or 'green hosting' here. Don't you care about that? I believe that Hetzner data centers in Europe (Germany, Finland) are powered by green energy, but not the locations in US.

huijzer 2y ago
Data centers used 460 TWh, or about 2% of total worldwide electricity use, according to IEA in 2022. In comparison, 30% of total energy (energy! Not electricity) goes to transport! As another point of comparison, transport in Sweden in 2022 used 137 TWh [1]. So the same order of magnitude as total datacenter energy use. And datacenters are powered by electricity which increases the chance that it comes from renewable energy. Conversely, the chance that diesel comes from a renewable source is zero. So can we please stop talking about data center energy use? It’s a narrative that the media is currently pushing but as so many things it makes no sense. It’s not the thing we should be focusing on if we want to decrease fossil fuel use. [1]: https://www.energimyndigheten.se/en/energysystem/energy-consumption/ https://www.energimyndigheten.se/en/energysystem/energy-cons...
- esher 2y ago
  Thanks for sharing. I care about it. I run a small hosting company. Sure, there are many low hanging fruits for fighting CO2 emissions that should be tackled first. I am also hopeful that energy from directly available renewables will be the most economic choice for building data centers anyhow, so that this is not a matter of believes any more. But on the other side, to bring down CO2 levels, fast change everywhere is required. As far as I see data center energy consumption continues to grow, specifically with AI. If I am not mistaken, data centers produce more CO2 than aviation. And sure, most 'green hosting' is probably 'green washing', yet I would still support and link initiatives such as: https://www.thegreenwebfoundation.org/ https://www.thegreenwebfoundation.org/
- davedx 2y ago
  2% of total worldwide electricity use in 2022 is a shit load of electricity and emissions. Your argument is the same as those who argue "our country shouldn't care about emissions when China is the biggest emitter". If you dive into a detailed breakdown of emissions you'll find that it's a complex hierarchy of categories. You can't just fix "all of transport" or treat it like a "low hanging fruit", just look at how much time it's taken for EV penetration to be in any way significant; look at how much of transport emissions are from aviation or shipping or other components. Any energy use that's measurable in whole percentage points of global emissions needs addressing. That includes data centers.

surrTurr 2y ago

> Hetzner volumes are, in my experience, too slow for a production database. While you may in the past have had a good experience running customer-facing databases on AWS EBS, with Hetzner's volumes we were seeing >50ms of IOWAIT with very low IOPS. See https://github.com/rook/rook/issues/14999 https://github.com/rook/rook/issues/14999 for benchmarks. I set up rook ceph on a talos k8s cluster (with vm volumes) and experienced similar low performance; however, I always thought that was because of the 1Gi vSwitch (i.e. networking problem)?! The SSD volumes were quite fast.

merpkz 2y ago
In my limited experience with rook-ceph it is strictly bare metal technology to deploy. On virtualization it will basically replicate your data to VM disks which usually are already replicated, so quite a bit of replication amplification will happen and tank your performance.
tehlike 2y ago
SSD volumes are physically on the same node, and afaik not redundant. The cloud vms are ceph clusters behind the scenes, and writes need to commit for 3+ machines. It's both network latency and inherent process latency Additionally, hetzner has an IOPS limit of 5000 and write limit of some amount that does not scale with the size of database. 50G has the same limits as 5TB. For this reason, people are sometimes using different table spaces in postgres for example. Ceph puts another burden on top of already-ceph-based cloud volumes, btw, so don't do that.
- p_l 2y ago
  RAID10 on local SSDs is pretty performant option, but yeah, it's per node.

mythz 2y ago

Been a happy Hetzner customer for over a decade, previously using their dedicated servers in their German DC's before migrating to their Cloud US VMs for better latency with the US. Slightly disappointed with their recent cuts of their generous 20TB free traffic down to 3TB (€1.19 per additional TB), but they still look to be a lot better value than all other US cloud providers we've evaluated. Whilst I wouldn't run Kubernetes by choice, we've had success moving our custom SSH / Docker compose deployments over to use GitHub Actions with kamal-deploy.org, easy to setup and nice UX tools for monitoring remote deployed apps [1] [1] https://servicestack.net/posts/kamal-deployments https://servicestack.net/posts/kamal-deployments

Voultapher 2y ago
Seems to be a US thing, maybe their peering partners are forcing them to raise prices, the German DC still stells the 20TB bandwidth https://www.hetzner.com/cloud/ https://www.hetzner.com/cloud/, but US is an order of magnitude less for the same price :/
- inemesitaffia 2y ago
  I don't see how traffic in Ashburn is more expensive than Frankfurt and Amsterdam. It's the sort of place where people say Transit is cheaper than paid peering. (For eyeball networks at least). I think carrying traffic from Europe for some images and videos might make sense financially. But there's always bulk CDN's
  - kuschku 2y ago
    > I don't see how traffic in Ashburn is more expensive than Frankfurt and Amsterdam. The vast majority of Hetzner's traffic in europe (and tbh, anyone's traffic) is free peering. Telekom is the one major exception.

bittermandel 2y ago

We're very happy to use Hetzner for our bare-metal staging environments to validate functionality, but I still feel reluctant to put our production there. Disks don't quite work as intended at all times and our vSwitch setup has gotten reset more than once. All of this makes sense considering the extremely low price.

adamcharnock 2y ago

We've [1] been using Hetzner's dedicated servers to provide Kubernetes clusters to our clients for a few years now. The performance is certainly excellent, we typically see request times half. And because the hardware is cheaper we can provide dedicated DevOps engineering time to each client. There are some caveats though: 1) A staging cluster for testing updates is really a must. YOLO-ing prod updates on a Sunday is no one's idea of fun. 2) Application level replication is king, followed by block-level replication (we use OpenEBS/Mayastor). After going through all the Postgres operators we found StackGres to (currently) be the best. 3) The Ansible playbooks are your assets. Once you have them down and well-commented for a given service then re-deploying that service in other cases (or again in the future) becomes straightforward. 4) If you can I'd recommend a dedicated 10G network to connect your servers. 1G just isn't quite enough when it comes to the combined load of prod traffic, plus image pulls, plus inter-service traffic. This also gives a 10x latency improvement over AWS intra-az. 5) If you want network redundancy you can create a 1G vSwitch (VLAN) on the 1G ports for internal use. Give each server a loopback IP, then use BGP to distribute routes (bird). 6) MinIO clusters (via the operator) are not that tricky to operate as long as you follow the well trodden path. This provides you with local high-bandwidth, low-latency object storage. 7) The initial investment to do this does take time. I'd put it at 2-4 months of undistracted skilled engineering time. 8) You can still push ancillary/annoying tasks off onto cloud providers (personally I'm a fan of CloudFlare for HTTP load balancing). [1]: https://lithus.eu https://lithus.eu

sureIy 2y ago
> I'd put it at 2-4 months of undistracted skilled engineering time. How much is that worth to your company/customer vs a higher monthly bill for the next 5 years? As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all, unless I expect a 10k/month AWS bill. xkcd comes to mind: https://xkcd.com/1319/ https://xkcd.com/1319/
- adamcharnock 2y ago
  > As a consultancy company, you want to sell that. As a customer, I don't see how that's worth it at all. Well I do rather agree, but as a consultancy I'm biased. But let's do some math. Say it's 4 months (because who has uninterrupted time), a senior rate of $1000/day. 20 days a month, so 80 days, is an $80k outlay. That's assuming you can get the skills (because AWS et al like to hire these kinds of engineers). Say one wants a 3 year payback, that is $2,200/month savings you need. Which seems highly achievable given some of the cloud spends I've seen, and that I think an 80-90% reduction in cloud spend is a good ballpark. The appeal of a consultancy is that we'll remove the up-front investment, provide the skills, de-risk the whole endeavour, even put engineers within your team, but you'll _only_ save 50%. The latter option is much more appealing in terms of hiring, risk, and cash-flow. But if your company has the skills, the cash, and the risk tolerance then maybe the former approach is best. EDIT: I actually think the(/our) consultancy option is a really good idea for startups. Their infrastructure ends up being slightly over-built to start with, but very quickly they end up saving a lot of money, and they also get DevOps staffing without having to hire for it. Moreover, the DevOps resource available to them scales with their compute needs. (also we offer 2x the amount of DevOps days for startups for the first year to help them get up and running).

mnming 2y ago

I feel lots of the work described in the article can be automated by kops, probably in a much better way, especially when it comes to day 2 operations. I wonder what is the motivation behind manually spinning up a cluster instead of going with more established tooling?

Hetzner_OL 2y ago

Hi Bill, Wow! Thanks for the amazing write-up and for sharing it on your blog and here! I am so happy that we've helped you save so much money and that you're happy with our support team! It's a great way to start off the week! --Katie

Neil44 2y ago

When I first started hosting servers/services for customers I was using EC2 and Rackspace, then I discovered Linode and was happy it was so much cheaper with apparently no downside. After the first couple of interactions with support I started to relax. Then I discovered OVH, same story. I haven't needed the support yet though.

wvh 2y ago

I work for a consultancy company that helps companies building and securing infrastructure. We have a lot of customers running Kubernetes at low-cost providers (like Hetzner), more local middle-tier and top-three (AWS, GCP, Azure). We also have some governmental, financial and medical companies that can not or will not run in public clouds, so they usually host on-prem. If Hetzner has an issue or glitch once a month, the middle-tier providers have one every 2-3 months, and a place like AWS maybe every 5-6 months. However, prices also follow that observation, so you have to carefully consider on a case-by-case basis whether adding some extra machines and backup and failure scenarios is a better deal. The major benefit by using basic hosting services is that their pricing is a lot more predictable; you pay for machines and scale as you go. Once you get hooked into all the extra services a provider like AWS provides, you might get some unexpectedly high bills and moving away might be a lot harder. For smaller companies, don't make short-sighted decisions that threaten your ability to survive long-term by choosing the easy solution or "free credits" scheme early on. There is no right answer here, just trade-offs.

jillesvangurp 2y ago

The key take home point here is not how amazingly cheap Hetzner is, which it is. But how much of an extortion game Google, Amazon, MS, etc. are playing with their cloud services. These are trillion dollar companies because they are raking in cash with extreme margins. Yes, there is some added value in the level of convenience provided. But maybe with a bit more competition, pricing could be more competitive. A lot more competitive.

acac10 2y ago

// Taking another slant at the discussion: Why kubernetes? Thank you for sharing your experience. I also have my 3 personal servers with Hetzner, plus a couple VM instances in Scaleways (French outfit). Disclaimer: I’m a Googler, was SRE for ~10 years for GMail, identity, social, apps (gsuites nowadays) and more, managed hundreds of jobs in Borg, one of the 3 founders of the current dev+devops internal platform (and I focused on the releases,prod,capacity side of the platform), dabbled in K8s on my personal time. My opinions, not Google’s. So, my question is: given the significant complexity that K8s brings (I don’t think anyone disputes this) why are people using it outside medium-large environments? There are simpler and yet flexible & effective job schedulers that are way easier to manage. Nomad is an example. Unless you have a LOT of machines to manage, with many jobs (I’d say +250) to manage, K8s complexity, brittleness and overhead are not justifiable, IMO. The emergence of tools like Terraform and the many other management layers in top of K8s that try to make it easier but just introduce more complexity and their own abstractions are in itself a sign of that inherent complexity. I would say that only a few companies in the world need that level of complexity. And then they will need it, for sure. But, for most is like buying a Formula 1 to commute in a city. One other aspect that I also noticed is that technical teams tend to carry on the mess they had in their previous “legacy” environment and just replicate in K8s, instead of trying to do an architectural design of the whole system needs. And K8s model enables that kind of mess: a “bucket of things”. Those two things combined, mean that nowadays every company has soaring cloud costs, are running things they know nothing about but are afraid to touch in case of breaking something. And an outage is more career harming than a high bill that Finance will deal with it later, so why risk it, right? A whole new IT area has been coined now to deal with this: FinOps :facepalm: I’m just puzzled by the whole situation, tbh.

KaiserPro 2y ago
I too used to run a large clustered environment (VFX) and now work at a FAANG which has a "borg-like" scheduler. K8s has a whole kit of parts which sound really grand when you are starting out on a new platform, but quickly become a pain when you actually start to implement it. I think thats the biggest problem, is by the time you've realised that actualy you don't need k8s, you've invested so much time into learning the sodding thing, its difficult to back out. The other seductive thing is helm provides "AWS-like" features (ie fancy load balancing rules) that are hard to figure out unless you've dabbled with the underlying tech before (varnish/nginx/etc are daunting, so is storage and networking) this tends to lead to utterly fucking stupid networking systems because unless you know better, that looks normal.

james_sulivan 2y ago

For those considering Hetzner, there is also Contabo which is another German hosting company that is also good, at least in my experience

0xbadcafebee 2y ago

I used to do my own car maintenance, because I wanted to save money, and it was fun. It turned out it was more complex than I thought, things slowly fell apart or broke. I spent a good deal of time "re-fixing" things. Spent probably thousands on tools over the years, partly replacing the cheap stuff that broke or rusted quickly. My cars were often up on blocks. But I learned a lot of great lessons. The biggest one? Some things are not worth DIYing; pay a mechanic or lease your car, especially if you depend on it for your livelihood. Even something as simple as an oil change, really isn't worth doing yourself. First you buy the tools (oil drip pan, filter wrench, funnel, creeper). Then you set aside the time to use them, find your dingy work clothes. You go to the store and buy new oil and a filter. You go home and change the oil. Then that day or another day you go to a store that will take your used oil. Versus 20 minutes at an auto mechanic, for about $15 more than the cost of the oil and filter. Kubernetes is an entire car (and a complex one). It's really not worth doing the maintenance yourself, I promise you. Unless you're just doing it for fun.

p_l 2y ago
Unless you find out that calling AA for everything and going to big name mechanics garage every time costs you significant portion of your budget. A lot of it is finding balance between what to do yourself, what to outsource, and it's not as easy or clean as some people here like to claim.
- infecto 2y ago
  +1 to this. I agree with the takeaway that paying for managed Kubernetes is the way to go 9 times out of 10 but I think in life its quite powerful to attempt to do things yourself sometimes. No, you should not go out and buy thousands of dollars worth of tools and do everything yourself but its pretty powerful to be able to do simple things in your life...change oil, replace light switches/sockets. The reason I personally find it powerful is that it helps me quantify what is worth outsourcing and what is not and it also builds mental models for how things work which is important when outsourcing work.
theappsecguy 2y ago
I don’t know. My shop wanted 1800 to change my brakes. I bought the parts for 300 and got it done in a day (first time). Seems like a pretty good payback and good skill to have. My neighbour has a car lift which certainly helped.

awinter-py 2y ago

cut my kube bill 100% on GKE by switching from regional to zonal cluster bc the first zonal cluster is free

deleted 2y ago

[deleted]

lucasrattz 2y ago

We at Syself.com also have great experiences with Kubernetes on Hetzner. We built a platform on top of Cluster API and brought a managed Kubernetes experience to Hetzner. Now we have self-healing, automated updates and 100% reproductibility, with full bare metal support. > Hetzner volumes are, in my experience, too slow for a production database. That's true, though. To solve that we developed a way to persist the local storage of bare metal servers across reprovisionings. This way it's both faster and cheaper. Now we are adding an automated database deployment layer on top of it.