23 ms·
Hetzner Servers Compromised
- anoother 13y agoFull text of the email sent to cutomers: Dear Client At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios). An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally. As a result, we currently have to consider the client data stored in our Robot as compromised. To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before. The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected. The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code. We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed. The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot. With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised. Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned. Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident. Naturally, we shall inform you of new developments immediately. We very much regret this incident and thank you for your understanding and trust in us. A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries. Kind regards Martin Hetzner
- pdpi 13y agoHuh, this doesn't make me happy. I haven't gotten an email yet.
- amarraja 13y agoHave all Hetzner customers received this mail? I currently have a couple of servers with them and have received nothing yet.
- moepstar 13y agoAFAIK yes - it might take a while though, i've gotten mine quite a while after the first ones seemed to have gotten them..
- sunsations 13y agoI also got the mail.
- PawelDecowski 13y agoI just got mine now (almost 45 minutes after this submission to HN) so yours is probably still in the queue.
- babuskov 13y agoI did not get it yet. Perhaps they are only sending to those who might be affected?
- mythz 13y agoyes, received mine this morning as well.
- amarraja 13y agoThanks all, finally got my mail. I was worried my account was hijacked! Panic over
- e12e 13y agoI can also confirm receiving the email (but I really need to tidy up my mail filters, I actually read about the hack here first). Nice email, one improvement would be to host the wiki behind a proper https-site -- especially since we already know the attack appears somewhat sophisticated, and that the attackers gained control over servers on the Hetzner network. Now, the faq doesn't contain anything obviously bad (email your credit card info to... etc) -- but one small thing that could be improved. Other than that I appreciate how Hetzner have handled this so far.
- tonetheman 13y agohmmm... what does Hetzner do?
- moepstar 13y agoThey're a German Server/Webspace ISP - see www.hetzner.de/en
- dspillett 13y agoThey are a provider of hosted dedicated servers, colo and virtual machines (with or without server management services), based in Germany.
- samspenc 13y agoThey're really inexpensive, which explains their niche popularity...
- Gravityloss 13y agoThey're very big, not just a niche. Also reliability / support is not necessarily always so much worse than some higher price solutions.
- dspillett 13y ago
- tbh 13y agoConsidering https://twitter.com/omgtbh/status/337567604887658496 https://twitter.com/omgtbh/status/337567604887658496 I can't say I'm surprised... (tweet text reproduced here: "I asked Hetzner if they plan to support 2 factor auth & was told that they already do - they require a username and a password. Seriously.")
- aw3c2 13y agoYou don't seriously think that a server admin said that, do you? It was probably some low rank customer service peon.
- tbh 13y agoOf course not, but regardless of who in the company said it, the official support response was as quoted. Not exactly encouraging!
- MrCheese 13y agoTo be fair, Hetzner has very "google translaty" English support for simple matters, so it is not impossible that they simply didn't understand the question
- PaulFreund 13y agoExactly, and somebody who didn't yet hear the term can easily come to the conclusion that it is an authentification with two "factors" which means basically two things and that would be username and password. In the past there were system with only one factor...
- 1SaltwaterC 13y agoKinda reminds me of http://www.youtube.com/watch?v=gmOTpIVxji8 http://www.youtube.com/watch?v=gmOTpIVxji8
- FooBarWidget 13y agoSigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually during boot. This way we're protected against many types of hosting provider hacks.
- pekk 13y agoWhat kind of effect does this have on your I/O performance?
- FooBarWidget 13y agoMinor. It costs more CPU because of encryption/decryption. But our servers have so much CPU power, and our workload is mostly dependent on disk I/O throughput, so enabling encryption is almost free.
- sneak 13y agoAES-NI features in modern CPUs reduce it even further.
- gkoz 13y agoWhat types of hacks does that protect you from exactly, considering you don't have physical access and can't even see the hardware?
- FooBarWidget 13y agoIt protects against hackers that compromise the provider's administration interface, and then use that interface to reboot the server to single user mode or a rescue system. Upon reboot, the attacker will encounter an unusable blob as filesystem because he doesn't have the encryption key. It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it. If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks. If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.
- X4 13y agoSeriously, again?? I think such a fauxpas shouldn't be tolerated twice.
- M4v3R 13y agoDo you have link to their previous hack?
- atesti 13y agoIn October 2011, unfortunately these links are in German only: http://www.heise.de/security/meldung/Web-Hoster-Hetzner-gehackt-1356501.html http://www.heise.de/security/meldung/Web-Hoster-Hetzner-geha... http://www.heise.de/ix/meldung/Datenleck-bei-Hetzner-1356468.html http://www.heise.de/ix/meldung/Datenleck-bei-Hetzner-1356468... http://www.heise.de/security/meldung/Passwortklau-bei-Hetzner-ueber-FTP-1358172.html http://www.heise.de/security/meldung/Passwortklau-bei-Hetzne... There was unauthorized access to customer data, even passwords. Someone claims to have accessed it via an FTP-server where he found a root password to a management server
- X4 13y agoThanks for posting the links, I've forgot to mention it in all my wisdom, duh..
- 925dk 13y agoHere is the (english) mail they sent to customers back in October 2011 on the previous intrusion: http://pastebin.com/uJhG5aLS http://pastebin.com/uJhG5aLS
- pekk 13y agoSeveral events with Linode, now Hetzner. These are relatively "premier," high-quality hosting companies, you can count on thousands and thousands of companies to pay even less attention. Yet every time, the discussion is only about one specific company, without seeing any broader pattern. When are we ever going to draw the conclusion that popular hosting companies (and, actually related, facilities like RubyGems) are especially attractive targets, and that the approach of waiting for an exploit and then shaming the targeted company is not an effective way of getting better security?
- gtaylor 13y agoThey are of good quality, but let's keep in mind that Hetzner tends to be solidly on the budget side. I am a Hetzner customer, and it stinks that this happened, but let's consider: I don't expect Bugatti quality when I'm paying for a Chevy Malibu. I do appreciate them being forthcoming and sharing some security details that do make me feel pretty safe. I don't care if they have the last few digits of my CC number, and I've already reset my password. Stinks I had to even think about this, but again, Chevy Malibu.
- TylerE 13y agoApply Occam's Razor. The simplest speculation is an internal breach. All the procedure in the world won't save you from that.
- hhw 13y agoAssuming that the exploit really is unknown, what better security could they have done to prevent it, without severely impacting usability? Compromises will happen sooner or later when running a large number of public facing services. With good policy, the breaches will be well contained, which seems to be the case here. What more can be expected out of a premier hosting company, let alone Hetzner which is very cheap budget provider?
- saturdayplace 13y agoIs it just the customer account details that apparently make hosting companies attractive targets. If that's the case, I'm wondering why we're not seeing more breaches from all over the e-commerce world. Why just hosting companies?
- jtchang 13y agoThe lesson to take from this is you can't rely on someone else providing security for you. You need to make sure you "layer" your security. If your database is compromised make sure you are salting your passwords or at least using a hash that is designed to be resistant against cracking. This doesn't protect against someone who has root access to your box and manages to capture passwords in the clear as they are being transferred but it does limit the surface of a breach.
- berkay 13y agoThings seem to be getting worse there. A server we have there just went offline, and their management servers are not responding. I wonder it's part of the exploit or as a result of attempts to "fix" the problem
- zapt02 13y agoI had 5 minutes of downtime, but everything seems to be back in working order.
- rebelde 13y agoSomething was happening with their routers, too. Using PingPlotter to one of my servers shows severe packet loss on the hos-bb2.juniper2.rz19.hetzner.de router. I see it just cleared up for now. Still, I'm moving my traffic to other locations till this blows over.
- PaulFreund 13y agoThat might be because various parts of germany are flooded at the moment. One of Hetzners datacenters is located in Falkenstein near Regensburg and others might me affected too
- illumen 13y agoTheir SSL certificate is fucked for their security page... oh dear. https://wiki.hetzner.de/index.php/Security_Issue/en https://wiki.hetzner.de/index.php/Security_Issue/en Hope everyone has fun with their identity theft cleanups. They ask people for ID scans. Bank details are compromised. boo! This was announced AFTER the German work day was done. Really nice for admins. Thanks.
- StavrosK 13y agoYou would have preferred that they waited until tomorrow?
- 925dk 13y agoAs part of the registration process with hetzner.de, you have to send them scans of personal documents (such as passport, drivers license or similar). I asked them just now if these systems were compromised and they promptly replied: "The system that stores scans of ids, credit cards and so on was not compromised. In addition to that, we delete that information after 21 days."
- larrys 13y ago"you have to send them scans of personal documents (such as passport, drivers license or similar)." What? Seriously? "In addition to that, we delete that information after 21 days." Hmm, do they delete the info that ends up on backup copies? How do you know they even actually delete it in 21 days? It's not like there is a third party even auditing which you can rely on. (Not that I'd ever do that for something like this anyway, I would just find another provider.)
- 925dk 13y agoYes seriously, here is what they asked me on first order with them: "Since you're a new customer with Hetzner, we ask you for a scan of your passport or ID card (authenticity check). It's only necessary for your first order with us. Please send the scan by fax or as an email attachment." When they say they delete it after 21 days, as they did in the mail I've just received, I trust them. I find their communication on this matter, as well as previous matters, open and serious.
- gokhan 13y agoThe info in the mail regarding the safety of credit card info contradicts with the linked FAQ. FAQ: Bank details are encrypted (two-way) in the database. However, it cannot be excluded that the attacker/s have also been able to obtain access to the key. Mail: With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
- antoko 13y agobanking details is a much broader term than credit card number. account numbers / routing numbers / company names / po boxes/ physical addresses / account history and other billing info. I'd assume they're saying some of these might be compromised, but specifically that credit cards aren't.
- perlgeek 13y agoHetzner probably has many customers who pay with debit card/recurring direct debit instead of credit card. It's quite common in Germany.
- thejosh 13y agoThey also allow PayPal, which I recently changed to with Hetzner. 99% of my payments to websites are PayPal, for this very reason.
- LogicX 13y agoPerhaps the email is different based on that particular customer's payment method, which is why one is talking about credit cards and the other about banking details?
- muxxa 13y agoI've got a login for robot.your-server.de and have changed my password but don't seem to have one for konsoleh.your-server.de. Is the konsoleh login/account something that would have gotten created for me automatically? i.e. do I need to worry about it?
- deleted 13y ago[deleted]
- ChuckMcM 13y agoOne of the differences between servers in data centers and desktops is that servers don't reboot a lot. So the bad guys can build an in memory system that doesn't change anything on disk (which avoids the configuration management system from flagging it) and if it cloaks itself as a normally long lived process then active monitors might be fooled as well. It then runs, effectively undetected, until the server reboots which can be months (or even years). Tools to protect against those threats are going to have start taking into account process activity and footprint.
- raylu 13y agoHetzner is very open, transparent, and exact in their communication about this. I'm not a customer but I really like that they didn't try to dress anything up and were forthcoming with information.
- chappi42 13y agoDoesn't surprise me. I'm a customer and the whole experience feels like this.
- sgt 13y agoI use the Robot to access a VNC console of my leased server. I can even do a CTRL-ALT-DELETE and boot the OS in single user mode from it, if I only have access to the Robot. Doesn't this imply that my server is potentially compromised as well?
- alexvr 13y agoAm I an overly-cautious nubcake, or do any of you also refrain from clicking the link when it says, "We've been compromised. Come visit our site!"? If I were a mean dude who hacked into some prominent site, the first thing I would do is submit a link to HN that points to a malicious "article" and laugh as malware gets thrust into thousands of computers.
- ricardobeat 13y agoWay too much mork. If you have a way of infecting computers by simply visiting a webpage, just tag it some kind of porn and post on reddit & other forums.