5 ms·

Not only that. It's single-DC, and Hetzner is somewhat known for having serious DC-wide network failures from time to time. The hardware used is consumer hardw

by Mo3 4y ago

Not only that. It's single-DC, and Hetzner is somewhat known for having serious DC-wide network failures from time to time. The hardware used is consumer hardware too. This is literally the opposite of what I would expect for a SaaS with uptime requirement... I feel like OP/the developers behind this might have made a really terrible choice and could be paying more in manpower than the savings they have by using a budget server discounter Don't get me wrong, I have nothing against Hetzner, but this seems like an unfit usecase. This is literally what you use cloud platforms for..

upupandup 4y ago
I didn't know that thanks for sharing. Has there been major cases recently? really puzzling that a healthcheck provider itself is not accounting for this potential issue. Is there some sort of edgy reverse proxy here? Doesn't seem like it. Even then there would be a big latency.
- Mo3 4y ago
  https://status.hetzner.com/ https://status.hetzner.com/ We haven't been using them for anything in production, so I am not 100% sure - they have a backbone outage right now however.. There are no edge / reverse proxies being described in the OP article, just a HAProxy configuration within the same DC.
  - upupandup 4y ago
    interesting....im also concerned that with the recent geopolitical developments, Hetzner could already be at crosshairs
    - Mo3 4y ago
      In case you are talking about potential DDoS attacks, yes, they have been hit hard by massive attacks the last few weeks. They also do not have adequate protection systems in place
- cuu508 4y ago
  About a month ago Hetzner was hit by an UDP DDOS, and as a mitigation they throttled UDP traffic on specific port ranges. This caused major problems for Healthchecks, as I'm using Wireguard for private networking, and Wireguard uses UDP. Here's a postmortem about this incident: https://status.healthchecks.io/en/incidents/dcbcDNd89LptHoWiUZ9iFK/ https://status.healthchecks.io/en/incidents/dcbcDNd89LptHoWi...
welterde 4y ago
It's not single-DC (since they have multiple locations and multiple datacenters at those locations), but they are a single provider. However AWS is also a single provider. And as Facebook and large ISPs have demonstrated recently-ish, having a global footprint does not mean you cannot break everything at the same time in interesting ways anyway. Only way around this is to go with multiple vendors, but now whatever you build to use them at the same time/fail over between them will be the weakest link. Hetzner also offers server-grade hardware btw. But if your setup is fully redundant anyway it doesn't really matter anymore if it's consumer grade or not (which the setup described in the article is).
- Mo3 4y ago
  We are discussing the service described in the article linked to with this post, which 1) runs single-DC 2) uses consumer hardware I agree on the notion that with redundancy it is not as important what kind of hardware is used, but it still is something to consider (seeing how this is a one-person venture and server outages are still a huge time-sink), and I wouldn't even necessarily speak about true redundancy in this case
  - welterde 4y ago
    The article explicitly said the servers are scattered across the various datacenters at the Falkenstein location - ergo not single-dc. Now you can argue about the amount of shared fate, but they are separate buildings and hetzner lists them as separate datacenters [0]. [0] https://docs.hetzner.com/general/others/data-centers-and-connection/ https://docs.hetzner.com/general/others/data-centers-and-con...
    - Mo3 4y ago
      Lol... okay, point taken, single-location. Not going to argue about the shared fate, as I've seen more outages than not affecting whole locations..
    - upupandup 4y ago
      im curious as to why they dont have US West, Central, Asia data centers. Even with a US West, Central offering they could really do a number with their pricing discounts. even better if they offer Singapore & Tokyo.
bckygldstn 4y ago
I run a one-person load-balanced SaaS on Hetzner. In the 2 years since I started I've had no customer-facing outages. Over the same period, I think there's been two outages of AWS us-east-1 and a multi-region GCP outage. Plus weekly outages of a grownup unicorn SaaS like github or slack. It's a tradeoff: the simplicity of such a setup reduces outage risk, and the costs saved can be spent on monitoring tools or extra redundancy.
Nextgrid 4y ago
Consumer hardware is totally fine if you use multiple machines - the probability of them failing at the exact same time is very low. His stack already uses multiple appservers (so the loss of one wouldn't even be noticeable) and has a manual way of failing over to a DB replica. It's often cheaper to work around node failures (which you often need to do for other reasons anyway - DC-wide outage, etc) than have more reliable nodes.

by Mo3 4y ago

upupandup 4y ago

I didn't know that thanks for sharing. Has there been major cases recently? really puzzling that a healthcheck provider itself is not accounting for this potential issue. Is there some sort of edgy reverse proxy here? Doesn't seem like it. Even then there would be a big latency.

Mo3 4y ago
https://status.hetzner.com/ https://status.hetzner.com/ We haven't been using them for anything in production, so I am not 100% sure - they have a backbone outage right now however.. There are no edge / reverse proxies being described in the OP article, just a HAProxy configuration within the same DC.
- upupandup 4y ago
  interesting....im also concerned that with the recent geopolitical developments, Hetzner could already be at crosshairs
  - Mo3 4y ago
    In case you are talking about potential DDoS attacks, yes, they have been hit hard by massive attacks the last few weeks. They also do not have adequate protection systems in place
cuu508 4y ago
About a month ago Hetzner was hit by an UDP DDOS, and as a mitigation they throttled UDP traffic on specific port ranges. This caused major problems for Healthchecks, as I'm using Wireguard for private networking, and Wireguard uses UDP. Here's a postmortem about this incident: https://status.healthchecks.io/en/incidents/dcbcDNd89LptHoWiUZ9iFK/ https://status.healthchecks.io/en/incidents/dcbcDNd89LptHoWi...

welterde 4y ago

It's not single-DC (since they have multiple locations and multiple datacenters at those locations), but they are a single provider. However AWS is also a single provider. And as Facebook and large ISPs have demonstrated recently-ish, having a global footprint does not mean you cannot break everything at the same time in interesting ways anyway. Only way around this is to go with multiple vendors, but now whatever you build to use them at the same time/fail over between them will be the weakest link. Hetzner also offers server-grade hardware btw. But if your setup is fully redundant anyway it doesn't really matter anymore if it's consumer grade or not (which the setup described in the article is).

Mo3 4y ago
We are discussing the service described in the article linked to with this post, which 1) runs single-DC 2) uses consumer hardware I agree on the notion that with redundancy it is not as important what kind of hardware is used, but it still is something to consider (seeing how this is a one-person venture and server outages are still a huge time-sink), and I wouldn't even necessarily speak about true redundancy in this case
- welterde 4y ago
  The article explicitly said the servers are scattered across the various datacenters at the Falkenstein location - ergo not single-dc. Now you can argue about the amount of shared fate, but they are separate buildings and hetzner lists them as separate datacenters [0]. [0] https://docs.hetzner.com/general/others/data-centers-and-connection/ https://docs.hetzner.com/general/others/data-centers-and-con...
  - Mo3 4y ago
    Lol... okay, point taken, single-location. Not going to argue about the shared fate, as I've seen more outages than not affecting whole locations..
  - upupandup 4y ago
    im curious as to why they dont have US West, Central, Asia data centers. Even with a US West, Central offering they could really do a number with their pricing discounts. even better if they offer Singapore & Tokyo.

bckygldstn 4y ago

I run a one-person load-balanced SaaS on Hetzner. In the 2 years since I started I've had no customer-facing outages. Over the same period, I think there's been two outages of AWS us-east-1 and a multi-region GCP outage. Plus weekly outages of a grownup unicorn SaaS like github or slack. It's a tradeoff: the simplicity of such a setup reduces outage risk, and the costs saved can be spent on monitoring tools or extra redundancy.

Nextgrid 4y ago

Consumer hardware is totally fine if you use multiple machines - the probability of them failing at the exact same time is very low. His stack already uses multiple appservers (so the loss of one wouldn't even be noticeable) and has a manual way of failing over to a DB replica. It's often cheaper to work around node failures (which you often need to do for other reasons anyway - DC-wide outage, etc) than have more reliable nodes.