6 ms·

Hetzner is really great until you try to scale with them. We started building our service on top of Hetzner and had couple 100s of VMs running and during peak t

Hetzner is really great until you try to scale with them. We started building our service on top of Hetzner and had couple 100s of VMs running and during peak time we had to scale them to over 1000 VMs. And here couple of problems started, you get pretty often IPs which are black listed, so if you try to connect to services hosted by Google, AWS like S3 etc. you can't reach them. Also at one point there were no VMs available anymore in our region, which caused a lot of issues. But in general if you don't need to scale crazy Hetzner is amazing, we still have a lot of stuff running on Hetzner but fan out to other services when we need to scale.

croes 1y ago
Blacklisted by whom?
- Hikikomori 1y ago
  AWS at least maintains IP lists of bots, active exploiters, ddos attackers, etc, that you can use to filter/rate limit traffic in WAF. Not so much AWS that blocks you but customers that decide to use these lists.
  - drcongo 1y ago
    Ironic, given how often the attacks I spend time fending off are coming from AWS.
  - croes 1y ago
    So AWS could list some IPs of competitors, just enough to make them look unreliable.
  - IshKebab 1y ago
    Nice IPs you've got there Hetzner, shame if...
- netdevphoenix 1y ago
  The big cloud providers I am assuming
CaptainOfCoit 1y ago
Worth noting that this seems to be about Hetzners cloud product, not the dedicated servers. The cloud product is relatively new, and most of the people who move to Hetzner do so because of the dedicated instances, not to use their cloud.
- drcongo 1y ago
  Hetzner's cloud offering is probably a decade old by now - I've been a very happy customer for 8 years.
  - CaptainOfCoit 1y ago
    You're right! I seem to have mixed it with some other dedi provider that added "cloud" recently. Thanks for the correction! My point of people moving to Hetzner for the dedicated instances rather than the cloud still remains though, at least in my bubble.
    - drcongo 1y ago
      No problem (I'm just glad you didn't read it as snark)! I mean, even 8 years is relatively new compared to their dedicated box offering so technically you were still correct.
  - watermelon0 1y ago
    Hetzner was founded in '97, so cloud offering could technically still be considered relatively new. :D
jakewins 1y ago
> Also at one point there were no VMs available anymore in our region, which caused a lot of issues. I'm not sure if this is a difference between other clouds, at least a few years ago this was a weekly or even daily problem in GCP; my experience is if you request hundreds of VMs rapidly during peak hours, all the clouds struggle.
- antonvs 1y ago
  We launch 30k+ VMs a day on GCP, regularly launching hundreds at a time when scheduled jobs are running. That’s one of the most stable aspects of our operation - in the last 5 years I’ve never seen GCP “struggle” with that except during major outages. At the scale of providers like AWS and even the smaller GCP, “hundreds of VMs” is not a large amount.
  - Macha 1y ago
    If you’re deploying something like 100 m5.4xlarge in us-east-1, sure, AWS’s capacity seems infinite. Once you get into high memory instances, GPU instances, less popular regions etc, it drops off. Now maybe after the AI demand and waves of purchases of systems appropriate for that things have improved, but it definitely wasn’t the case at the large scale employer I worked at in 2023 (my current employer is much smaller, so doesn’t have those needs, so I can’t comment)
  - dinvlad 1y ago
    Not a single VM possible to request on Azure us-east for over a month now though :-(
    - antonvs 1y ago
      I was talking about real cloud providers :P
- GordonS 1y ago
  I don't use Azure much anymore, but I used to see this problem regularly on Azure too, especially in the more "niche" regions like UK South.
jwr 1y ago
Note that we might be talking about two different things here: some of us use physical servers from Hetzner, which are crazy fast, and a great value. And some of us prefer virtual servers, which (IMHO) are not that revolutionary, even though still much less expensive than the competition.
jgalt212 1y ago
1000 VMs? So you have approx 1MM concurrent customers? That's a big number. You should definitely be able to get preferred pricing from AWS at that scale.
- breadislove 1y ago
  We have extremely processing heavy jobs where user upload large collection of files (PDFs, audios, videos etc.) and expect to get fast processing.
FBISurveillance 1y ago
We scaled to ~1100 bare metal servers with them and it worked perfectly.
- atonse 1y ago
  Username checks out.
matt-p 1y ago
I think they're great but it's unfortunate they don't have more locations which would at least enable you to spin VMs up in different locations during a shortage. If you rely on them it might be wise to have a second cloud provider that you can use in a pinch, there's many options.
V__ 1y ago
This sound really intriguing, and I am really curious. What kind of service do you run where you need a 100s of VMs? Was there a reason for not going dedicated? Looking at their offering their biggest VM is (48 CPU, 192 GB RAM, 960 GB SSD). I can't even imagine using that much. Again, I'm really curious.
- breadislove 1y ago
  we have extremely processing heavy jobs where user upload large collection of files (audios, pdfs, videos etc.) and expect to get fast processing. its just that we need to fan out sometimes, since a lot of our users a sensitive to processing times.
jamesblonde 1y ago
The blocking of services on Hetzner and Scaleway by Microsoft is well known - https://www.linkedin.com/posts/jeroen-jacobs-8209391_something-interesting-happened-today-it-activity-7382774062164516864-GSKT/ https://www.linkedin.com/posts/jeroen-jacobs-8209391_somethi... I didn't know AWS and GCP also did it. Not surprised. The problem is that European regulators do nothing about such anti-competitive dirty tricks. The big clouds hide behind "lots of spam coming from them", which is not true.
- lossyalgo 1y ago
  First comment on that post claims that according to Mimecast, 37% of EU-based spam originates from Hetzner and Digital Ocean. People have been asking for 3 days for a link to the source (I can't find it either). On the other hand, someone linked a report from last year[0]: > 72% of BEC attacks in Q2 2024 used free webmail domains; within those, 72.4% used Gmail. Roughly ~52% of all BEC messages were sent from Gmail accounts that quarter. [0] https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf
GordonS 1y ago
I've ran into the IP deny list problem too, but for Windows VMs - you spin them up, only to realise that you can't get Windows Updates, can't reach the Powershell gallery etc. And just deleting it and starting again is just going to give you the exact same IP again! I ended up having to buy a dozen or so IPs until I found one that wasn't blocked, and then I could delete all the blocked ones.

croes 1y ago

Blacklisted by whom?

Hikikomori 1y ago
AWS at least maintains IP lists of bots, active exploiters, ddos attackers, etc, that you can use to filter/rate limit traffic in WAF. Not so much AWS that blocks you but customers that decide to use these lists.
- drcongo 1y ago
  Ironic, given how often the attacks I spend time fending off are coming from AWS.
- croes 1y ago
  So AWS could list some IPs of competitors, just enough to make them look unreliable.
- IshKebab 1y ago
  Nice IPs you've got there Hetzner, shame if...
netdevphoenix 1y ago
The big cloud providers I am assuming

CaptainOfCoit 1y ago

Worth noting that this seems to be about Hetzners cloud product, not the dedicated servers. The cloud product is relatively new, and most of the people who move to Hetzner do so because of the dedicated instances, not to use their cloud.

drcongo 1y ago
Hetzner's cloud offering is probably a decade old by now - I've been a very happy customer for 8 years.
- CaptainOfCoit 1y ago
  You're right! I seem to have mixed it with some other dedi provider that added "cloud" recently. Thanks for the correction! My point of people moving to Hetzner for the dedicated instances rather than the cloud still remains though, at least in my bubble.
  - drcongo 1y ago
    No problem (I'm just glad you didn't read it as snark)! I mean, even 8 years is relatively new compared to their dedicated box offering so technically you were still correct.
- watermelon0 1y ago
  Hetzner was founded in '97, so cloud offering could technically still be considered relatively new. :D

jakewins 1y ago

> Also at one point there were no VMs available anymore in our region, which caused a lot of issues. I'm not sure if this is a difference between other clouds, at least a few years ago this was a weekly or even daily problem in GCP; my experience is if you request hundreds of VMs rapidly during peak hours, all the clouds struggle.

antonvs 1y ago
We launch 30k+ VMs a day on GCP, regularly launching hundreds at a time when scheduled jobs are running. That’s one of the most stable aspects of our operation - in the last 5 years I’ve never seen GCP “struggle” with that except during major outages. At the scale of providers like AWS and even the smaller GCP, “hundreds of VMs” is not a large amount.
- Macha 1y ago
  If you’re deploying something like 100 m5.4xlarge in us-east-1, sure, AWS’s capacity seems infinite. Once you get into high memory instances, GPU instances, less popular regions etc, it drops off. Now maybe after the AI demand and waves of purchases of systems appropriate for that things have improved, but it definitely wasn’t the case at the large scale employer I worked at in 2023 (my current employer is much smaller, so doesn’t have those needs, so I can’t comment)
- dinvlad 1y ago
  Not a single VM possible to request on Azure us-east for over a month now though :-(
  - antonvs 1y ago
    I was talking about real cloud providers :P
GordonS 1y ago
I don't use Azure much anymore, but I used to see this problem regularly on Azure too, especially in the more "niche" regions like UK South.

jwr 1y ago

Note that we might be talking about two different things here: some of us use physical servers from Hetzner, which are crazy fast, and a great value. And some of us prefer virtual servers, which (IMHO) are not that revolutionary, even though still much less expensive than the competition.

jgalt212 1y ago

1000 VMs? So you have approx 1MM concurrent customers? That's a big number. You should definitely be able to get preferred pricing from AWS at that scale.

breadislove 1y ago
We have extremely processing heavy jobs where user upload large collection of files (PDFs, audios, videos etc.) and expect to get fast processing.

FBISurveillance 1y ago

We scaled to ~1100 bare metal servers with them and it worked perfectly.

atonse 1y ago
Username checks out.

matt-p 1y ago

I think they're great but it's unfortunate they don't have more locations which would at least enable you to spin VMs up in different locations during a shortage. If you rely on them it might be wise to have a second cloud provider that you can use in a pinch, there's many options.

V__ 1y ago

This sound really intriguing, and I am really curious. What kind of service do you run where you need a 100s of VMs? Was there a reason for not going dedicated? Looking at their offering their biggest VM is (48 CPU, 192 GB RAM, 960 GB SSD). I can't even imagine using that much. Again, I'm really curious.

breadislove 1y ago
we have extremely processing heavy jobs where user upload large collection of files (audios, pdfs, videos etc.) and expect to get fast processing. its just that we need to fan out sometimes, since a lot of our users a sensitive to processing times.

jamesblonde 1y ago

The blocking of services on Hetzner and Scaleway by Microsoft is well known - https://www.linkedin.com/posts/jeroen-jacobs-8209391_something-interesting-happened-today-it-activity-7382774062164516864-GSKT/ https://www.linkedin.com/posts/jeroen-jacobs-8209391_somethi... I didn't know AWS and GCP also did it. Not surprised. The problem is that European regulators do nothing about such anti-competitive dirty tricks. The big clouds hide behind "lots of spam coming from them", which is not true.

lossyalgo 1y ago
First comment on that post claims that according to Mimecast, 37% of EU-based spam originates from Hetzner and Digital Ocean. People have been asking for 3 days for a link to the source (I can't find it either). On the other hand, someone linked a report from last year[0]: > 72% of BEC attacks in Q2 2024 used free webmail domains; within those, 72.4% used Gmail. Roughly ~52% of all BEC messages were sent from Gmail accounts that quarter. [0] https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf https://docs.apwg.org/reports/apwg_trends_report_q2_2024.pdf

GordonS 1y ago

I've ran into the IP deny list problem too, but for Windows VMs - you spin them up, only to realise that you can't get Windows Updates, can't reach the Powershell gallery etc. And just deleting it and starting again is just going to give you the exact same IP again! I ended up having to buy a dozen or so IPs until I found one that wasn't blocked, and then I could delete all the blocked ones.