6 ms·

Hetzner Private Cloud

60 points by stemuk 9y ago

merb 9y ago
I've also stumbled upon that. However the question remains, how to add more compute nodes? Is there a easy process to do that? How about updating it? Isn't it carless to actually say: You will have sole and unrestricted administration rights to the dedicated hardware with root access. Hetzner Online will not have access to the servers, and will therefore not be able to provide server administration support. ? I mean if I can't provision it, it's probably problematic to update it.
- Terretta 9y ago
  You are right, it's not so much private cloud on bare metal as bulk metal you manage. OTOH, if that's what you want (many enterprises think they do), this positioning and landing page puts Hetzner into consideration.
tribby 9y ago
the auction servers seem like better bang for the buck
- stemuk 9y ago
  True, but for the sake of easy management I would still imagine the OpenStack setup to be better. Hopefully they will add some configuration options that make it possible to choose every server type in their lineup to be configured as OpenStack cloud.
- tempay 9y ago
  But they're aimed at individuals who want to buy something cheaply. Larger business customers will want to be able to say we want X of Y machine and will pay the premium to get it.
- pudo 9y ago
  We've noticed a larger-than-normal number of disk failures on second-hand auctioned systems. Be sure to get one with a fresh, "enterprise-grade" drive.
- catdog 9y ago
  Depends on what you want. It's used (for many years usually) Hardware which increases the probability of failure and you only get whatever single configuration is currently available. It's nice though that they offer you to cancel at no charge for 14 days and the variety of configurations available which are often more asymmetric compared to their usual offerings, e.g. if you primarily need a lot of disk space you don't have to pay for a lot of RAM and a fast CPU.
  - chrisper 9y ago
    That doesn't sound like an issue though if you just need something cheap for now to just get going. You can keep in mind that it could fail any time and just plan around that (for now).
chridal 9y ago
This really needs a website that has a more modern feel to it. It doesn't exactly evoke trust.
- 55555 9y ago
  They are one of the largest server hosts in the world.
  - chridal 9y ago
    that doesn't have anything to do with my comment.
    - throwaway2048 9y ago
      if you care more about websites with a "modern feel". I'm sure companies like amazon and google are more than happy to charge you 10x more money for it.
      - ryanlol 9y ago
        Chridal is right though, there's lots of SQLi bugs on Hetzner. Not something you get on AWS or GCP. Hetzners website looks like an old shitty PHP hack because it very much is one.
- tyingq 9y ago
  Check out Berkshire Hathaway's website. http://www.berkshirehathaway.com http://www.berkshirehathaway.com
  - chridal 9y ago
    You've really done a great job when the site explicitly has to state "Official Website"...
zyztem 9y ago
Desktop processor without ECC memory plus very slow SATA hard drives - not a setup for reliability and performance
- dragonsh 9y ago
  I don't get it for 90% of startup this setup is more than enough to start and far cheaper then goog, Amazon, azure. Just the problem is most modern devops are lazy to learn the basics of setting a HA system with proper backup. Hertzner make it easy but one needs to know what they are doing. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).
  - HalfwayToDice 9y ago
    most modern devops are lazy to learn the basics of setting a HA system Absurd nonsense.
  - brianwawok 9y ago
    So what, I save $100 over Google but spend 3h a week futzing with configs?
    - kuschku 9y ago
      I’ve run the numbers with Hetzner, GCP, AWS, and even more managed products such as Firebase, Heroku, etc. Bare Metal compared to GCP or AWS will save you about 75% of your operating costs. Bare Metal compared to Heroku or Firebase will save you around 90% of your operating costs (a large part of this being caused by bandwidth, which is massively overpriced at Google and Amazon). With those savings you can usually serve 5 to 10 times as many customers. That might not be worth it if you’re in SV, and pay your devs the same wages Amazon or Google do (as then you’ll just pay more than with AWS or GCP), but if you’re in places where you pay half the wages Amazon or Google do4 , you can actually save a lot with this arrangement.
tyingq 9y ago
This isn't quite hitting the spot to me (non-server hardware, you still administer the OS, single location), but it does seem indicative of a potential new wave of low-cost cloud providers. Someone like Vultr, Linode, or DO is positioned well to deliver an "AWS lite" offering. They all have decent hardware, lots of locations, and a good delivery history. A bit of work to put together ELB/EC2/Lambda/S3 equivalents and a control panel would open up a new market. Especially if they offered low egress pricing.
- dragonsh 9y ago
  For me it's on spot, I don't like Amzn, goog, msft blackbox. This is a decent HA system for 90% startup adding just backup will make it perfect. It's better then ELB/EC2/Lambda/S3 as it uses openstack api under your own control. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).
  - tyingq 9y ago
    I just don't see how it's any different than what they had yesterday. You can buy two cheap non-server-hardware servers and put openstack or coreos on them and administer it yourself. All they did here was pre-install openstack, but leave you to keep it updated. If you can't install it yourself, good luck upgrading it when a security release comes out. I mentioned ELB/EC2/Lambda/S3 equivalents for 2 reasons. First, AWS is clearly the market share leader, so having a similar pattern might get more buyers. Second, it's a bit easier pattern for apps that aren't cloud aware. But, they could just offer hosted K8S with some add-on ingress controllers.
    - catdog 9y ago
      I also don't get what this offers over renting these machines individually. It can't really be some preinstalled software with no support whatsoever as installing usually is the easy part.
ollybee 9y ago
It's totally unsupported. If you were not able to just rent the tin and configure this yourself then you're probably not going to be able to maintain it long term. This product is dangerous and some people will learn that too late.
- stemuk 9y ago
  As far as I understood it the OpenStack software will be automatically updated on the releases of the LTS channel, so calling the product dangerous seems like an exaggeration to me.
  - simplehuman 9y ago
    Have you run openstack? There are million dollar companies out there that just support maintaining any updating openstack for a reason...
    - tyingq 9y ago
      Their list of pieces and parts gives a good insight to how complicated the beast is: https://www.openstack.org/software/project-navigator#tiles https://www.openstack.org/software/project-navigator#tiles
  - iamacynic 9y ago
    auto-updating openstack on production systems people don't know how to run themselves sounds pretty much like the most dangerous thing you could possibly come up with. why not just hand your systems over to a smart 15 year old kid to run, and go take a nice beachside vacation in the meantime? probably would work out better.
  - praneshp 9y ago
    I worked in/on Openstack for 3.5 years. It's next to impossible to run it without patches, so it's questionable if they can do auto-updates on releases of the LTS channel. Someone needs to sit down and migrate patches and remove any breakages.
- 9y ago
therealmarv 9y ago
OpenStack. Way overkill. I would advice always against OpenStack unless you have very good reasons to go into this direction.
breakingcups 9y ago
Here I was enthusiastic, but it's not-quite-there. If they took up administration of the Openstack setup this would be a very interesting product.
mark_l_watson 9y ago
I have been happy with using Hetzner services in the past and I wish I had a business use for this. I used openstack for a while on IBM bluemix and found it convenient enough. I would like a good backup recovery setup. I hope that they configure the systems for some redundancy between the two servers, but it is not clear from the ordering page.
thinkindie 9y ago
The company I'm working for moved away from hetzner coz it has a single location and far too many noisy neighbourhood. Hetzner was often target of DDoS and our connectivity was also affected.
- deleted 9y ago
  [deleted]
Scotrix 9y ago
We have a lot of experience with Hetzner servers and I used and recommend this hoster since at least 10 years intensively with tens and hundreds of servers. The most important thing with Hetzner Servers is to monitor everything very closely: - CPU Temperature - RAM - Disks (SMART) - Software- and Hardware raids - Network (interface) errors The servers are usually consumer-grade hardware components which have more often issues under heavy load so you have to expect down-times and broken components. However, if you are aware of that and you can easily shift around that with your software Hetzner will save you serious money (10 to 15 times cheaper than GCP and AWS). Also to mention is that their customer support is first class if you tell them all required details and exactly what to do. Usually they respond in minutes and do hardware replacements within an hour and small downtimes.
- stemuk 9y ago
  What do you think of their more 'server-like' offerings like the PX61 line [0]? What do I have to expect in terms of downtime and component replacements? [0] https://www.hetzner.de/de/hosting/produkte_rootserver/px61nvme https://www.hetzner.de/de/hosting/produkte_rootserver/px61nv...
  - chrisper 9y ago
    Those are actually (I think Fujitsu) Workstations. You should expect the same downtime you would expect from your own workstation if you were to use one at home. Maybe a bit less downtime since there would be less startups and the temperature differences would be smaller as well. Obviously, server grade hardware will perform and last longer under heavy loads. I believe if you don't go with the Server auction thing, then you get quite new hardware. So I guess you would not really have to expect a lot of downtime. But as always, downtimes can happen. Always plan accordingly.
    - Scotrix 9y ago
      Absolutely, the new servers are ok but after a while of permanent load (a year or so), you'll also have to expect issues, at least in our experience. The server auction itself is worse but they replace hardware without any issues and most often the replaced hardware is new as well.
bonjurkes 9y ago
I used Hetzner like 10 years ago and didn't had good experience. I doubt anything changed over time. They offer home grade hardware and low priority support in return of cheap prices. They only deal with hardware and network issues (which is quite normal for Dedi service with normal support option) - Hardware they offer is more prone to fail because of using home grade hardware for long time (especially HDD). - It's almost impossible to convince them HDD is failing even with showing SMART logs. Hardware needs to fail so they will replace it. - Hardware replacement times are quite fast (thanks to SLA). They replace it with another used HDD, if you want something newer, than they ask some money for replacing with less used HDD. - They scan their network regularly for hosted malware, trojan etc. so if one of your sites get hijacked and has iframe viruses etc. Hetzner will null route your server. - If your IP gets DDOS, null route. - If you get DMCA warning, null route without waiting 24 hours. - If your NAT leaks your internal traffic to WLAN, null route. - It takes almost few day to lift null route ban on your server when you get in contact with support. It's okay for support tickets to wait in queue for long time because of service level but I believe null route tickets needs priority no matter what. We decided to move over to another provider after having problems. Hetzner also owns few other brands like Serverloft.
- angry-hacker 9y ago
  DMCA 24h?! It means I could 'troll' someone out of business doing bogus complaints? At least out of their server.
- fulafel 9y ago
  Why would a German company do DMCA?
  - samoa4 9y ago
    good question ... another german hosting provider also required you to at least respond to dmca + three strikes
- samoa4 9y ago
  > Hetzner also owns few other brands like Serverloft. "serverloft ist ein Produkt der Host Europe GmbH." https://www.serverloft.de/unternehmen/agb/ https://www.serverloft.de/unternehmen/agb/ doesnt say so on the english website thou
msmm 9y ago
But it is in germany - sort of police state in term of Internet. No thanks.
- kakoni 9y ago
  In Autumn 2017 there's going to be a new Hetzner datacenter in Finland [0]. [0] https://www.hetzner.de/it/hosting/presse/info-0916 https://www.hetzner.de/it/hosting/presse/info-0916
pella 9y ago
Hetzner: Dedicated Root Servers AMD Ryzen : https://www.hetzner.de/us/hosting/produktmatrix/rootserver-amd https://www.hetzner.de/us/hosting/produktmatrix/rootserver-a... * AMD Ryzen 5 1600X * AMD Ryzen 7 1700X
sanbor 9y ago
I used Hetzner services until I received an email from there about some of their servers being infected in the RAM. Here is the email that I received in 2013: Dear Client At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios). An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally. As a result, we currently have to consider the client data stored in our Robot as compromised. To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before. The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected. The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code. We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed. The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot. With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised. Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned. Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident. Naturally, we shall inform you of new developments immediately. We very much regret this incident and thank you for your understanding and trust in us. A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries. Kind regards Martin Hetzner

60 points by stemuk 9y ago

merb 9y ago

I've also stumbled upon that. However the question remains, how to add more compute nodes? Is there a easy process to do that? How about updating it? Isn't it carless to actually say: You will have sole and unrestricted administration rights to the dedicated hardware with root access. Hetzner Online will not have access to the servers, and will therefore not be able to provide server administration support. ? I mean if I can't provision it, it's probably problematic to update it.

Terretta 9y ago
You are right, it's not so much private cloud on bare metal as bulk metal you manage. OTOH, if that's what you want (many enterprises think they do), this positioning and landing page puts Hetzner into consideration.

tribby 9y ago

the auction servers seem like better bang for the buck

stemuk 9y ago
True, but for the sake of easy management I would still imagine the OpenStack setup to be better. Hopefully they will add some configuration options that make it possible to choose every server type in their lineup to be configured as OpenStack cloud.
tempay 9y ago
But they're aimed at individuals who want to buy something cheaply. Larger business customers will want to be able to say we want X of Y machine and will pay the premium to get it.
pudo 9y ago
We've noticed a larger-than-normal number of disk failures on second-hand auctioned systems. Be sure to get one with a fresh, "enterprise-grade" drive.
catdog 9y ago
Depends on what you want. It's used (for many years usually) Hardware which increases the probability of failure and you only get whatever single configuration is currently available. It's nice though that they offer you to cancel at no charge for 14 days and the variety of configurations available which are often more asymmetric compared to their usual offerings, e.g. if you primarily need a lot of disk space you don't have to pay for a lot of RAM and a fast CPU.
- chrisper 9y ago
  That doesn't sound like an issue though if you just need something cheap for now to just get going. You can keep in mind that it could fail any time and just plan around that (for now).

chridal 9y ago

This really needs a website that has a more modern feel to it. It doesn't exactly evoke trust.

55555 9y ago
They are one of the largest server hosts in the world.
- chridal 9y ago
  that doesn't have anything to do with my comment.
  - throwaway2048 9y ago
    if you care more about websites with a "modern feel". I'm sure companies like amazon and google are more than happy to charge you 10x more money for it.
    - ryanlol 9y ago
      Chridal is right though, there's lots of SQLi bugs on Hetzner. Not something you get on AWS or GCP. Hetzners website looks like an old shitty PHP hack because it very much is one.
tyingq 9y ago
Check out Berkshire Hathaway's website. http://www.berkshirehathaway.com http://www.berkshirehathaway.com
- chridal 9y ago
  You've really done a great job when the site explicitly has to state "Official Website"...

zyztem 9y ago

Desktop processor without ECC memory plus very slow SATA hard drives - not a setup for reliability and performance

dragonsh 9y ago
I don't get it for 90% of startup this setup is more than enough to start and far cheaper then goog, Amazon, azure. Just the problem is most modern devops are lazy to learn the basics of setting a HA system with proper backup. Hertzner make it easy but one needs to know what they are doing. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).
- HalfwayToDice 9y ago
  most modern devops are lazy to learn the basics of setting a HA system Absurd nonsense.
- brianwawok 9y ago
  So what, I save $100 over Google but spend 3h a week futzing with configs?
  - kuschku 9y ago
    I’ve run the numbers with Hetzner, GCP, AWS, and even more managed products such as Firebase, Heroku, etc. Bare Metal compared to GCP or AWS will save you about 75% of your operating costs. Bare Metal compared to Heroku or Firebase will save you around 90% of your operating costs (a large part of this being caused by bandwidth, which is massively overpriced at Google and Amazon). With those savings you can usually serve 5 to 10 times as many customers. That might not be worth it if you’re in SV, and pay your devs the same wages Amazon or Google do (as then you’ll just pay more than with AWS or GCP), but if you’re in places where you pay half the wages Amazon or Google do4 , you can actually save a lot with this arrangement.

tyingq 9y ago

This isn't quite hitting the spot to me (non-server hardware, you still administer the OS, single location), but it does seem indicative of a potential new wave of low-cost cloud providers. Someone like Vultr, Linode, or DO is positioned well to deliver an "AWS lite" offering. They all have decent hardware, lots of locations, and a good delivery history. A bit of work to put together ELB/EC2/Lambda/S3 equivalents and a control panel would open up a new market. Especially if they offered low egress pricing.

dragonsh 9y ago
For me it's on spot, I don't like Amzn, goog, msft blackbox. This is a decent HA system for 90% startup adding just backup will make it perfect. It's better then ELB/EC2/Lambda/S3 as it uses openstack api under your own control. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).
- tyingq 9y ago
  I just don't see how it's any different than what they had yesterday. You can buy two cheap non-server-hardware servers and put openstack or coreos on them and administer it yourself. All they did here was pre-install openstack, but leave you to keep it updated. If you can't install it yourself, good luck upgrading it when a security release comes out. I mentioned ELB/EC2/Lambda/S3 equivalents for 2 reasons. First, AWS is clearly the market share leader, so having a similar pattern might get more buyers. Second, it's a bit easier pattern for apps that aren't cloud aware. But, they could just offer hosted K8S with some add-on ingress controllers.
  - catdog 9y ago
    I also don't get what this offers over renting these machines individually. It can't really be some preinstalled software with no support whatsoever as installing usually is the easy part.

ollybee 9y ago

It's totally unsupported. If you were not able to just rent the tin and configure this yourself then you're probably not going to be able to maintain it long term. This product is dangerous and some people will learn that too late.

stemuk 9y ago
As far as I understood it the OpenStack software will be automatically updated on the releases of the LTS channel, so calling the product dangerous seems like an exaggeration to me.
- simplehuman 9y ago
  Have you run openstack? There are million dollar companies out there that just support maintaining any updating openstack for a reason...
  - tyingq 9y ago
    Their list of pieces and parts gives a good insight to how complicated the beast is: https://www.openstack.org/software/project-navigator#tiles https://www.openstack.org/software/project-navigator#tiles
- iamacynic 9y ago
  auto-updating openstack on production systems people don't know how to run themselves sounds pretty much like the most dangerous thing you could possibly come up with. why not just hand your systems over to a smart 15 year old kid to run, and go take a nice beachside vacation in the meantime? probably would work out better.
- praneshp 9y ago
  I worked in/on Openstack for 3.5 years. It's next to impossible to run it without patches, so it's questionable if they can do auto-updates on releases of the LTS channel. Someone needs to sit down and migrate patches and remove any breakages.
9y ago

therealmarv 9y ago

OpenStack. Way overkill. I would advice always against OpenStack unless you have very good reasons to go into this direction.

breakingcups 9y ago

Here I was enthusiastic, but it's not-quite-there. If they took up administration of the Openstack setup this would be a very interesting product.

mark_l_watson 9y ago

I have been happy with using Hetzner services in the past and I wish I had a business use for this. I used openstack for a while on IBM bluemix and found it convenient enough. I would like a good backup recovery setup. I hope that they configure the systems for some redundancy between the two servers, but it is not clear from the ordering page.

thinkindie 9y ago

The company I'm working for moved away from hetzner coz it has a single location and far too many noisy neighbourhood. Hetzner was often target of DDoS and our connectivity was also affected.

deleted 9y ago
[deleted]

Scotrix 9y ago

We have a lot of experience with Hetzner servers and I used and recommend this hoster since at least 10 years intensively with tens and hundreds of servers. The most important thing with Hetzner Servers is to monitor everything very closely: - CPU Temperature - RAM - Disks (SMART) - Software- and Hardware raids - Network (interface) errors The servers are usually consumer-grade hardware components which have more often issues under heavy load so you have to expect down-times and broken components. However, if you are aware of that and you can easily shift around that with your software Hetzner will save you serious money (10 to 15 times cheaper than GCP and AWS). Also to mention is that their customer support is first class if you tell them all required details and exactly what to do. Usually they respond in minutes and do hardware replacements within an hour and small downtimes.

stemuk 9y ago
What do you think of their more 'server-like' offerings like the PX61 line [0]? What do I have to expect in terms of downtime and component replacements? [0] https://www.hetzner.de/de/hosting/produkte_rootserver/px61nvme https://www.hetzner.de/de/hosting/produkte_rootserver/px61nv...
- chrisper 9y ago
  Those are actually (I think Fujitsu) Workstations. You should expect the same downtime you would expect from your own workstation if you were to use one at home. Maybe a bit less downtime since there would be less startups and the temperature differences would be smaller as well. Obviously, server grade hardware will perform and last longer under heavy loads. I believe if you don't go with the Server auction thing, then you get quite new hardware. So I guess you would not really have to expect a lot of downtime. But as always, downtimes can happen. Always plan accordingly.
  - Scotrix 9y ago
    Absolutely, the new servers are ok but after a while of permanent load (a year or so), you'll also have to expect issues, at least in our experience. The server auction itself is worse but they replace hardware without any issues and most often the replaced hardware is new as well.

bonjurkes 9y ago

I used Hetzner like 10 years ago and didn't had good experience. I doubt anything changed over time. They offer home grade hardware and low priority support in return of cheap prices. They only deal with hardware and network issues (which is quite normal for Dedi service with normal support option) - Hardware they offer is more prone to fail because of using home grade hardware for long time (especially HDD). - It's almost impossible to convince them HDD is failing even with showing SMART logs. Hardware needs to fail so they will replace it. - Hardware replacement times are quite fast (thanks to SLA). They replace it with another used HDD, if you want something newer, than they ask some money for replacing with less used HDD. - They scan their network regularly for hosted malware, trojan etc. so if one of your sites get hijacked and has iframe viruses etc. Hetzner will null route your server. - If your IP gets DDOS, null route. - If you get DMCA warning, null route without waiting 24 hours. - If your NAT leaks your internal traffic to WLAN, null route. - It takes almost few day to lift null route ban on your server when you get in contact with support. It's okay for support tickets to wait in queue for long time because of service level but I believe null route tickets needs priority no matter what. We decided to move over to another provider after having problems. Hetzner also owns few other brands like Serverloft.

angry-hacker 9y ago
DMCA 24h?! It means I could 'troll' someone out of business doing bogus complaints? At least out of their server.
fulafel 9y ago
Why would a German company do DMCA?
- samoa4 9y ago
  good question ... another german hosting provider also required you to at least respond to dmca + three strikes
samoa4 9y ago
> Hetzner also owns few other brands like Serverloft. "serverloft ist ein Produkt der Host Europe GmbH." https://www.serverloft.de/unternehmen/agb/ https://www.serverloft.de/unternehmen/agb/ doesnt say so on the english website thou

msmm 9y ago

But it is in germany - sort of police state in term of Internet. No thanks.

kakoni 9y ago
In Autumn 2017 there's going to be a new Hetzner datacenter in Finland [0]. [0] https://www.hetzner.de/it/hosting/presse/info-0916 https://www.hetzner.de/it/hosting/presse/info-0916

pella 9y ago

Hetzner: Dedicated Root Servers AMD Ryzen : https://www.hetzner.de/us/hosting/produktmatrix/rootserver-amd https://www.hetzner.de/us/hosting/produktmatrix/rootserver-a... * AMD Ryzen 5 1600X * AMD Ryzen 7 1700X

sanbor 9y ago

I used Hetzner services until I received an email from there about some of their servers being infected in the RAM. Here is the email that I received in 2013: Dear Client At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios). An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally. As a result, we currently have to consider the client data stored in our Robot as compromised. To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before. The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected. The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code. We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed. The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot. With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised. Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned. Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident. Naturally, we shall inform you of new developments immediately. We very much regret this incident and thank you for your understanding and trust in us. A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries. Kind regards Martin Hetzner