5 ms·

I am running a cluster of machines at Hetzner for years. Generally i am satisfied, but i like to share some bad experiences which took a month to fix. At some p

I am running a cluster of machines at Hetzner for years. Generally i am satisfied, but i like to share some bad experiences which took a month to fix. At some point some of my servers were facing random cpu stalls mostly at occuring midnight. I spend hours and nights to find out the cause. But i wasnt able to find it. Tracing the issue resulting in different causes everytime. After I contacted the Hetzner support team they moved some of the servers to a different host system. Apparently there was a resource issue in their virtualization layer. It fixed all the issues for a week, but then it started again. I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further. I was perplexed. I solved the issue by creating servers in a different zone. Exact copies of the so called faulty ones. The whole stack is now running without issues for about 2 months. But still i am a bit worried.

leoff 4y ago
Something similar happened to me. A server I have was very frequently randomly restarting, and I tried everything to solve it, i.e factory reset, reinstalling my services. I contacted customer support and they offered me to take it offline and perform some tests on it that could take more than 10 hours. After I insisted a lot, they offered me a one time server replacement, which magically fixed the restarting issue.
- Hetzner_OL 4y ago
  Thanks for writing about how our team helped you resolve the issue. Have you been happy with the rest of your experiences using the cloud products? --Katie
  - darkstar_16 4y ago
    That's not what we wrote. The support team replaced the server after a lot of cajoling. That experience needs to be fixed.
    - deleted 4y ago
      [deleted]
  - leoff 4y ago
    You thinking this was a positive experience only makes me more concerned about your customer support. Let's hope I don't need another "one-time" server replacement because of your faulty servers, because I already used mine. 10 hours of downtime with angry customers on my tail sounds fun.
Kiro 4y ago
And why are you so sure it's not on your end?
- sph 4y ago
  > I solved the issue by creating servers in a different zone. Exact copies of the so called faulty ones.
  - Kiro 4y ago
    Considering the issue didn't start happening immediately the previous times I don't see why we can presume it won't start happening again this time.
    - nextstepguy 4y ago
      - hardware2win 4y ago
        What even is this approach?
Hetzner_OL 4y ago
Perhaps you could give me a ticket number (or the latest ticket number) about this issue, and I can ask a team member to review it for you. Or, you can send the ticket number directly to marketing@hetzner.com with a link to this page, and I can do the same thing. --Katie
- Traubenfuchs 4y ago
  High visibility customer support via hacker news is not a feature, it‘s a failure. And those failed communications between provider and customer are why everyone is frustrated with the big corpos (google, cloudflare, stripe, etc.) which Hetzner is apparently bound to become.
  - spaniard89277 4y ago
    Support is a very hard issue to solve when you have volume. It doesn't help that 99.9% companies regard support as a cost center so salaries are low and there's a lot of churn and disregard for supportpeople tools.
  - TeMPOraL 4y ago
    At least you have someone speaking for Hetzner officially here. Contrast that with Google, where Matt Cutts was kind of used as unofficial backchannel for customer complaints, up until he left Google, and now all we have are prayers that some unknown reader will be moved by our plights. This is not to excuse bad support experiences. I just think we should appreciate it when there is still a channel to a real human being willing to be helpful and engage constructively, because it's getting increasingly rare these days.
  - rocqua 4y ago
    Isn't it great that they recognize this failure and are attempting to fix it?
    - aflukasz 4y ago
      It's not the first time I read a feedback about unprofessional customer support from Hetzner. Here or on Reddit. I admit that nearly every time someone from Hetzner shows up, says it might me a language/cultural issue or something of this kind and offer direct help to look into it. Hetzner is on my radar for a long time as an alternative to simpler deployments on AWS. It's great they are monitoring internet forums and try to be helpful, but also very frustrating to read such problems apparently still occur.
RicoElectrico 4y ago
> I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further. Doesn't surprise me, German companies always know better and will try to prove you wrong, whether you're their customer or supplier. Such is the business culture, I think.
- unity1001 4y ago
  > Doesn't surprise me, German companies always know better and will try to prove you wrong Yep. Its a plight that afflicts almost all engineers around the world, and German companies are even more afflicted by it due to the German engineering culture...
- Hetzner_OL 4y ago
  As someone who has customer service experience in both the USA and Germany, I can tell you that there are, indeed, cultural differences. There are certain situations in the USA where Germans and other Europeans don't always have the best experience because of a cultural misunderstanding. I am also the in-house English teacher here at Hetzner. (Most companies have no in-house teacher.) Something that I personally work on with my students in my conversations class and customer service class is intonation, which can cause spoken language to come off as sounding "arrogant" by accident. In addition, written responses may accidentally come of as sounding too direct or "arrogant" for the same reason. We work on these situations in my classes. So if you ever have a ticket that you think might make for good learning material for one of my classes, or that you would like to see escalated because of a serious language/cultural misunderstanding, please write to marketing@hetzner.com and mention my name (and include the relevant ticket number). --Katie
  - elkos 4y ago
    While it might be a cultural issue, I have to admit that this isn't the case with many Germans. I interact daily in open-source projects I'm involved with. While they are direct most of them seem very respectful and welcoming. From time to time some people might seem a bit more arrogant but I think that there might be a cultural bias too at play here. Kudos for assisting your colleagues with better communication in English, I think this practice could be useful for many organizations with people that speak English as a foreign language.
tinco 4y ago
We had a similar problem at Hetzner, we were using their consumer grade systems and this was over 6 years ago. To Hetzner's credit though, after we decided it had to be a hardware problem (dmesg would log something about cpu states before every crash) we sent in a ticket and within a reasonable timeframe (I think within a day or a couple, it was long ago so don't know for sure) a Hetzner sysadmin went into the BIOS and changed some feature (I suppose he disabled some sleep mode) and the systems ran perfect after that. I chalked it up to our decision to run our databases on consumer grade hardware and didn't give it a second thought.
PaulKeeble 4y ago
I had a weird performance issues on Hetzner virtual machines where network bandwidth would drop to 50mbit/s like it was being throttled. My application had a tendency to use very little bandwidth most of the time but when updates were available we needed 10-40GB in a day or so and I always suspected we were getting throttled but support never admitted it. On average we were below their resource limits but clearly at times we were above the average. Support was pretty dismissive and arrogant and I don't regret moving off to netcup who in comparison the experience has been flawless.
jeremyjh 4y ago
Are you using their “standard” cloud or dedicated vcpu product?
unity1001 4y ago
> I contacted support again and received an arrogant email Yep. The attitude of the engineering support is something that Hetzner needs to improve a LOT on. They tend to treat customers as if they are treating members of their own open source or engineering community - scolding the customers when they think the customer (community member) is being unreasonable etc. Its a great thing that eng. support is just a ticket and 20-45 minutes away at Hetzner. But the attitude needs a lot of improvement. Imagine that you are a startup facing some quirky issue that affects your business in the middle of the night and having to deal with attitude and scolds from datacenter engineers...

leoff 4y ago

Something similar happened to me. A server I have was very frequently randomly restarting, and I tried everything to solve it, i.e factory reset, reinstalling my services. I contacted customer support and they offered me to take it offline and perform some tests on it that could take more than 10 hours. After I insisted a lot, they offered me a one time server replacement, which magically fixed the restarting issue.

Hetzner_OL 4y ago
Thanks for writing about how our team helped you resolve the issue. Have you been happy with the rest of your experiences using the cloud products? --Katie
- darkstar_16 4y ago
  That's not what we wrote. The support team replaced the server after a lot of cajoling. That experience needs to be fixed.
  - deleted 4y ago
    [deleted]
- leoff 4y ago
  You thinking this was a positive experience only makes me more concerned about your customer support. Let's hope I don't need another "one-time" server replacement because of your faulty servers, because I already used mine. 10 hours of downtime with angry customers on my tail sounds fun.

Kiro 4y ago

And why are you so sure it's not on your end?

sph 4y ago
> I solved the issue by creating servers in a different zone. Exact copies of the so called faulty ones.
- Kiro 4y ago
  Considering the issue didn't start happening immediately the previous times I don't see why we can presume it won't start happening again this time.
  - nextstepguy 4y ago
    - hardware2win 4y ago
      What even is this approach?

Hetzner_OL 4y ago

Perhaps you could give me a ticket number (or the latest ticket number) about this issue, and I can ask a team member to review it for you. Or, you can send the ticket number directly to marketing@hetzner.com with a link to this page, and I can do the same thing. --Katie

Traubenfuchs 4y ago
High visibility customer support via hacker news is not a feature, it‘s a failure. And those failed communications between provider and customer are why everyone is frustrated with the big corpos (google, cloudflare, stripe, etc.) which Hetzner is apparently bound to become.
- spaniard89277 4y ago
  Support is a very hard issue to solve when you have volume. It doesn't help that 99.9% companies regard support as a cost center so salaries are low and there's a lot of churn and disregard for supportpeople tools.
- TeMPOraL 4y ago
  At least you have someone speaking for Hetzner officially here. Contrast that with Google, where Matt Cutts was kind of used as unofficial backchannel for customer complaints, up until he left Google, and now all we have are prayers that some unknown reader will be moved by our plights. This is not to excuse bad support experiences. I just think we should appreciate it when there is still a channel to a real human being willing to be helpful and engage constructively, because it's getting increasingly rare these days.
- rocqua 4y ago
  Isn't it great that they recognize this failure and are attempting to fix it?
  - aflukasz 4y ago
    It's not the first time I read a feedback about unprofessional customer support from Hetzner. Here or on Reddit. I admit that nearly every time someone from Hetzner shows up, says it might me a language/cultural issue or something of this kind and offer direct help to look into it. Hetzner is on my radar for a long time as an alternative to simpler deployments on AWS. It's great they are monitoring internet forums and try to be helpful, but also very frustrating to read such problems apparently still occur.

RicoElectrico 4y ago

> I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further. Doesn't surprise me, German companies always know better and will try to prove you wrong, whether you're their customer or supplier. Such is the business culture, I think.

unity1001 4y ago
> Doesn't surprise me, German companies always know better and will try to prove you wrong Yep. Its a plight that afflicts almost all engineers around the world, and German companies are even more afflicted by it due to the German engineering culture...
Hetzner_OL 4y ago
As someone who has customer service experience in both the USA and Germany, I can tell you that there are, indeed, cultural differences. There are certain situations in the USA where Germans and other Europeans don't always have the best experience because of a cultural misunderstanding. I am also the in-house English teacher here at Hetzner. (Most companies have no in-house teacher.) Something that I personally work on with my students in my conversations class and customer service class is intonation, which can cause spoken language to come off as sounding "arrogant" by accident. In addition, written responses may accidentally come of as sounding too direct or "arrogant" for the same reason. We work on these situations in my classes. So if you ever have a ticket that you think might make for good learning material for one of my classes, or that you would like to see escalated because of a serious language/cultural misunderstanding, please write to marketing@hetzner.com and mention my name (and include the relevant ticket number). --Katie
- elkos 4y ago
  While it might be a cultural issue, I have to admit that this isn't the case with many Germans. I interact daily in open-source projects I'm involved with. While they are direct most of them seem very respectful and welcoming. From time to time some people might seem a bit more arrogant but I think that there might be a cultural bias too at play here. Kudos for assisting your colleagues with better communication in English, I think this practice could be useful for many organizations with people that speak English as a foreign language.

tinco 4y ago

We had a similar problem at Hetzner, we were using their consumer grade systems and this was over 6 years ago. To Hetzner's credit though, after we decided it had to be a hardware problem (dmesg would log something about cpu states before every crash) we sent in a ticket and within a reasonable timeframe (I think within a day or a couple, it was long ago so don't know for sure) a Hetzner sysadmin went into the BIOS and changed some feature (I suppose he disabled some sleep mode) and the systems ran perfect after that. I chalked it up to our decision to run our databases on consumer grade hardware and didn't give it a second thought.

PaulKeeble 4y ago

I had a weird performance issues on Hetzner virtual machines where network bandwidth would drop to 50mbit/s like it was being throttled. My application had a tendency to use very little bandwidth most of the time but when updates were available we needed 10-40GB in a day or so and I always suspected we were getting throttled but support never admitted it. On average we were below their resource limits but clearly at times we were above the average. Support was pretty dismissive and arrogant and I don't regret moving off to netcup who in comparison the experience has been flawless.

jeremyjh 4y ago

Are you using their “standard” cloud or dedicated vcpu product?

unity1001 4y ago

> I contacted support again and received an arrogant email Yep. The attitude of the engineering support is something that Hetzner needs to improve a LOT on. They tend to treat customers as if they are treating members of their own open source or engineering community - scolding the customers when they think the customer (community member) is being unreasonable etc. Its a great thing that eng. support is just a ticket and 20-45 minutes away at Hetzner. But the attitude needs a lot of improvement. Imagine that you are a startup facing some quirky issue that affects your business in the middle of the night and having to deal with attitude and scolds from datacenter engineers...