5 ms·
As a data point, I've been running stuff at Hetzner for 10 years now, in two datacenters (physical servers). There were brief network outages when they replaced
by jwr 1y ago
As a data point, I've been running stuff at Hetzner for 10 years now, in two datacenters (physical servers). There were brief network outages when they replaced networking equipment, and exactly ONE outage for hardware replacement, scheduled weeks in advance, with a 4-hour window and around 1-2h duration.
It's just a single data point, but for me that's a pretty good record.
It's not because Hetzner is miraculously better at infrastructure, it's because physical servers are way simpler than the extremely complex software and networking systems that AWS provides.
- jamesbelchamber 1y ago> physical servers are way simpler than the extremely complex software and networking systems that AWS provides. Or, rather, it's your fault when the complex software and networking systems you deployed on top of those physical servers go wrong (:
- jwr 1y agoYes. Which is why I try to keep my software from being overly complex, for example by not succumbing to the Kubernetes craze.
- brianwawok 1y agoI love my k8s. Spend 5 minutes per month over the past 8 years and get a very reliable infra
- cj 1y agoDo you work on k8s professionally outside of the project you’re talking about? 5 mins seems unrealistic unless you’re spending time somewhere else to keep up to speed with version releases, upgrades, etc.
- dangus 1y agoI think it sounds quite realistic especially if you’re using something like Talos Linux. I’m not using k8s personally but the moment I moved from traditional infrastructure (chef server + VMs) to containers (Portainer) my level of effort went down by like 10x.
- nullify88 1y agoI would say even if not using Talos, Argo CD or Flux CD together with Renovate really helps to simplify the reoccuring maintenence.
- datadrivenangel 1y agoYou've spent less than 8 hours total on kubernetes?
- margorczynski 1y agoWell the complexity comes not from Kubernetes per se but that the problem it wants to solve (generalized solution for distributed computing) is very hard in itself.
- alephnil 1y agoOnly if you actually has a system complex enough to require it. A lot of systems that use kubernetes are not complex enough to require it, but use it anyway. In that case kubernetes does indeed add unnecessary complexity.
- otabdeveloper4 1y agoExcept that k8s doesn't solve the problem of generalized distributed computing at all. (For that you need distributed fault-tolerant state handling which k8s doesn't do.) K8s solves only one problem - the problem of organizational structure scaling. For example, when your Ops team and your Dev team have different product deadlines and different budgets. At this point you will need the insanity of k8s.
- naikrovek 1y agoI am so happy to read that someone views kubernetes the same way I do. for many years i have been surrounded by people who "kubernetes all the things" and that is absolute madness to me.
- cutler 1y agoYes, I remember when Kubernetes hit the scene and it was only used by huge companies who needed to spin-up fleets of servers on demand. The idea of using it for small startup infra was absurd.
- scaryclam 1y agoAs another data point, I run a k8s cluster on Hetzner (mainly for my own experience, as I'd rather learn on my pet projects vs production), and haven't had any Hetzner related issues with it. So Hetzner is OK for the overly complex as well, if you wish to do so.
- gamaralf 1y agoI agree. Even when Kubernetes is used in large environments, is it still cumbersome, verbose and overly complex.
- tester54654 1y agoWhat are the alternatives?
- tester54654 1y agoRight, who needs scalability? Each app should have a hard limit of users and just stop acceppting new users when limits are reached.
- sgarland 1y agoYou know that “scale” existed long before K8s - or even Borg - was a thing, right? I mean, how do you think Google ran before creating them?
- tester54654 1y agoyes and mobile phones existed before smartphones, what's the point? So far in terms of scalability nothing beats k8s. And from OpenAI and Google we also see that it even works for high performance use case such as LLM trainings with huge amounts of nodes.
- SJC_Hacker 1y agoYeah scalability is great! Let’s burn through thousands of dollars an hour and give all our money to Amazon/Google/Microsoft When those pink slips come in, we’ll just go somewhere else and do the same thing!
- tete 1y agoIf the complex software you deployed and/or configured goes wrong on AWS it's also your fault.
- sapiogram 1y agoDo you monitor your product closely enough to know that there weren't other brief outages? E.g. something on the scale of unscheduled server restarts, and minute-long network outages?
- supriyo-biswas 1y agoI personally do through status monitors at larger cloud providers at 30 sec resolutions, never noticed a downtime. They will sometimes drop ICMP though, even though the host is alive and kicking.
- firesteelrain 1y agoSurprised they allow ICMP at all
- naikrovek 1y agowhy does this surprise you? actually, why do people block ICMP? I remember in 1997-1998 there were some Cisco ICMP vulnerabilities and people started blocking ICMP then and mostly never stopped, and I never understood why. ICMP is so valuable for troubleshooting in certain situations.
- firesteelrain 1y agoI don’t disagree. I am used to highly regulated industries where ping is blocked across the WAN
- scrps 1y agoSecurity through obscurity mostly, I don't know who continues to push the advice to block ICMP without a valid technical reason since at best if you tilt your head and squint your eyes you could almost maybe see a (very new) script kiddie being defeated by it. I've rarely actually seen that advice anywhere, more so 20 years ago than now but people are still clearly getting it from circles I don't run in.
- 1y ago
- yread 1y ago7 years, 20 servers, same here.
- elktown 1y agoCloud marketing and career incentives seems to have instilled in the average dev that MTBF for hardware is in days rather than years.
- cutler 1y agoMTBF?
- Sanzig 1y agoMean Time Between Failures.
- oceansweep 1y agoMean time between failures
- saynay 1y agoOn the other hand, I had the misfortune of having a hardware failure on one of my Hetzner servers. They got a replacement harddrive in fairly quickly, but still complete data loss on that server, so I had to rebuild it from scratch. This was extra painful, because I wasn't using one of the OS that is blessed by Hetzner, so it requires a remote install. Remote installs require a system that can run their Java web plugin, and that have a stable and fast enough connection to not time out. The only way I have reliably gotten them to work is by having an ancient Linux VM that was also running in Hetzner, and had the oldest Firefox version I could find that still supported Java in the browser. My fault for trying to use what they provide in a way that is outside their intended use, and props to them for letting me do it anyway.
- jwr 1y agoThat can happen with any server, physical or virtual, at any time, and one should be prepared for it. I learned a long time ago that servers should be an output of your declarative server management configuration, not something that is the source of any configuration state. In other words, you should have a system where you can recreate all your servers at any time. In your case, I would indeed consider starting with one of the OS base installs that they provide. Much as I dislike the Linux distribution I'm using now, it is quite popular, so I can treat it as a common denominator that my ansible can start from.
- kro 1y agoThey allow netbooting to a recovery OS from which the disks can be provisioned via an ssh session too, for custom setups. Likely there are cases that require the remote "keyboard", but I wanted to mention that.