7 ms·
Sigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually du
by FooBarWidget 13y ago
Sigh, another hosting provider hack. As if Linode and OVH are not enough. This is the reason why we use full disk encryption, where we enter the key manually during boot. This way we're protected against many types of hosting provider hacks.
- pekk 13y agoWhat kind of effect does this have on your I/O performance?
- FooBarWidget 13y agoMinor. It costs more CPU because of encryption/decryption. But our servers have so much CPU power, and our workload is mostly dependent on disk I/O throughput, so enabling encryption is almost free.
- sneak 13y agoAES-NI features in modern CPUs reduce it even further.
- gkoz 13y agoWhat types of hacks does that protect you from exactly, considering you don't have physical access and can't even see the hardware?
- FooBarWidget 13y agoIt protects against hackers that compromise the provider's administration interface, and then use that interface to reboot the server to single user mode or a rescue system. Upon reboot, the attacker will encounter an unusable blob as filesystem because he doesn't have the encryption key. It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it. If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks. If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.
- gkoz 13y agoVery good point about single user/rescue, thanks.
- jonknee 13y agoDo you have a very small number of servers? Requiring manual steps to boot is not very scalable--if there is an event that requires all your servers to reboot (power outage, mandatory upgrade, etc) you would be in quite a tough spot.
- FooBarWidget 13y agoI much prefer having to repeatedly enter the password dozens of times than risking data leaking out - or worse, data being tampered with.
- adw 13y agoSo "yes" – what would you do if you had, say, several hundred (or several thousand) servers?
- FooBarWidget 13y agoIf I have that many servers then I can assume I have a large budget for security, right? I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part. But I'm not a security expert. Maybe I've overlooked something.
- raylu 13y agoWait, this sounds way harder than the previous solution and entering the password over a remote IPMI console.
- FooBarWidget 13y ago
- epochwolf 13y agoHow does this help? The key is still stored in memory which I assume the hypervisor has access to.
- FooBarWidget 13y agoCorrect, but hacking into the hypervisor is harder than hacking an administration interface. In the end the only secure server is one in a vault at the bottom of the ocean, but there are ways to prevent certain attack vectors.
- yk 13y ago>In the end the only secure server is one in a vault at the bottom of the ocean[...] I strongly doubt that: https://en.wikipedia.org/wiki/Glomar_Explorer https://en.wikipedia.org/wiki/Glomar_Explorer
- tbh 13y agoHetzner provide a lot of physical machines too, I believe this is what the other poster was talking about.
- FooBarWidget 13y agoPhysical machines don't prevent keys from leaking out. A physical attacker can analyze power usage usage patterns to extract the encryption key. :)
- mike_esspe 13y agoWhat if attacker infects unencrypted part used for booting? Do you have protection against that? It's relatively easy on Linux, where /boot is usually not encrypted.
- scott_karana 13y agoAgreed, but typically /boot isn't automounted when it's a separate partition, which means they would need a root-access compromise already.
- claudius 13y agoThey can replace the booting kernel. Unless you use SecureBoot, TPM or some such thing, there is no way to protect against that (assuming that the attacker has access to the shut-down system at one point and you boot it later).
- mike_esspe 13y agoHetzner allows to netboot any server via admin panel. After booting into recovery os, you can modify unencrypted parts as you wish.
- FooBarWidget 13y agoThe trick is putting all important data in the encrypted part. The unencrypted part must be reconstructable in an automated manner. If you are paranoid and you assume that all unplanned reboots are attacks, then you can reconstruct the unencrypted part every time. Tools like Chef make this relatively easy.
- sneak 13y agoThe kernel lives on /boot, and could be subverted to lie about its uptime to convince you that there was no unplanned reboot. This is the same methodology I use, and I think about these attacks a lot. There is no good/cheap way to verify a remote execution environment right now with commodity hardware. :/
- e12e 13y agoWell, maybe. Do you have servers with a secure boot environment? Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices). As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard. I still think that it mostly defends against someone stealing/cloning the physical disks though.
- FooBarWidget 13y agoEven without secure boot environment, if you're paranoid enough you can throw away and reconstruct the boot environment on every unplanned boot. Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.
- e12e 13y agoMy point was that getting back to a known good boot scenario isn't all that easy if you don't have physical access? I suppose the sanest middle ground is to wait for the dust of an event like this to settle (or have someone investigate, in case of "just" an unexpected reboot) -- then wipe the server(s) in question, reinstall (hoping the install media is ok) -- and importing the disks/backup. I still think it is little gain for a lot of work. Basically if your server software is secure, you've avoided data compromise due to a reboot into single user mode or similar -- but you have to assume (for the sake of guaranteeing the safety of the encrypted data) -- that attackers have a copy of the data at rest (which they can get if the can boot into a recovery image etc. So now you have to make sure that you can boot a safe environment in which to import the data to, without leaking your key(s). All the while, it is rather likely there are a few vulnerabilities in your software stack that all this never did mitigate. I'm not saying don't do it -- I'm in favour of doing it for the (relative) peace of mind that no that will get lost/out as the servers are recycled -- I'm not sure it makes much sense to avoid hackers -- the online threat seems to dominate for most cases I can think of.