11 ms·

Problems with disk encryption in AWS

44 points by DrRobinson 4y ago

andorov 4y ago
What about the disk being reused by another process without being thoroughly wiped by AWS?
- colonelpopcorn 4y ago
  That's an interesting idea. I imagine they don't do DoD wipes of their disks because they are probably multi-tenant, right? Can a guest operating system do forensic analysis on the host disk?
  - britneybitch 4y ago
    AWS wipes disks before reusing them (loosely speaking--an EBS volume doesn't correspond 1:1 to a physical disk). Forensic analysis of overwritten data requires disassembling the physical disk, so no, it's not possible.
- whoknew1122 4y ago
  Disks are wiped immediately before they are made available as a new volume. If your policies require a certain level of wipe, you should wipe the disk before releasing it.
likecarter 4y ago
Stealing disks does happen, especially on-premise, which is where these policies originated. Also, sometimes there are mistakes with decommissioning old drives, and you wouldn’t want your data discovered in a landfill somewhere.
- eloff 4y ago
  The author is talking specifically about AWS. The odds that there is a mistake decommissioning the disk that leaves the data intact, times that somebody salvaged it from a landfill, times that they care about your data is basically zero. Which means a logical person should worry about everything else.
  - im3w1l 4y ago
    What are the odds that some arranges for all those things to happen though? When you try to go after them they will have plausible deniability.
    - eloff 4y ago
      That’s an interesting attack vector. Bribe someone to replace the disk without wiping it, and be in a position to intercept it after that.
      - DrRobinson 4y ago
        This would require you or the person in the data center to know which customer is using which disk. And data isn't stored on just one disk, it's spread out over multiple disks and many customers have shards of data stored on the same disk. So even if this did happen, the would only get fragments of data.
    - fnordpiglet 4y ago
      Disk management and destruction is largely automated. That’s extraordinarily unlikely.
  - knorker 4y ago
    You forgot to multiply by the millions of disks they presumably go through every year.
    - eloff 4y ago
      You don’t though. Your risk calculation is the same.
- rolenthedeep 4y ago
  I once bought a "new" hard drive off Amazon. The connectors looked suspiciously scratched up, like it had been used before. When I dug deeper, they had wiped the SMART data and the partition table, but it was absolutely full of readable data. I found clear text server logs indicating that this drive was in a backblaze center for several thousand hours.
  - dexterdog 4y ago
    Did you report that to backblaze? I would think a bug bounty would be paid on something like that.
    - Jensson 4y ago
      I'd worry they would sue for hacking to cover it up, dumber things has happened to nice developers who report vulnerabilities.
      - brianwski 4y ago
        Disclaimer: I work at Backblaze, and I was here first. > I'd worry they would sue for... If you are referring to Backblaze, we're not going to "sue" anybody for anything. We (Backblaze) have dealt with a bunch of frivolous lawsuits (and patent trolls) over the years suing us, and OMG we're not going to instigate any lawsuits over some honest person legitimately reporting some issue and being helpful. It isn't going to happen. Our reputation is important to us. Not just for Backblaze: I'm saying the individuals that founded Backblaze and those people that work now here base our entire existence and careers and the number one marketing efforts at Backblaze are based around we are trying to be "the good guys" and transparent and acting like it. There is no possibly world where we try to suppress a screwup like this through legal means. That would be a PR debacle of epic proportions. If something went wrong, let's shine a spotlight on that cockroach and figure it out together. I'm not sure the exact drive we are all talking about, but my first guess would be a customer ordered a $189 "USB Restore" all their data shipped to them on an encrypted drive) and we (Backblaze) shipped the customer a USB restore drive and they are subsequently selling it (after copying their restore off of it) on the open market. If it is above 8 TBytes this is absolutely *NOT* the case and we should get to the bottom of it. Without lawyers mucking up the situation.
- kube-system 4y ago
  They don't go into a landfill unless they're broken. Old drives that someone could still get the data off of end up on eBay.
- lokar 4y ago
  Google has very tight physical security (metal detectors, etc) and disks are either confirmed erased or shredded I assume aws is similar
  - LammyL 4y ago
    Google mostly runs their own data centers. Amazon mostly rents space in commercial data centers with lots of different companies. Aws is probably pretty secure but likely less physically secure compared to Google if you get into the nitty gritty details.
    - fnordpiglet 4y ago
      This is false. Aws builds and operates their own data centers. The design and architecture of their data centers are highly standardized and an enormous amount of the build is fully automated. When they do lease they lease just a basic power and fiber build and they build their data center ontop of the base. Given how much custom equipment they run at the data center level it would be impractical to do anything that was collocated. But most data centers for aws, especially in Virginia, are on Amazon owned land. As far as I am aware the only collocations are in the satellite zones, outposts, and in peering locations. https://www.datacenterknowledge.com/archives/2017/01/18/who-leased-the-most-data-center-space-in-2016 https://www.datacenterknowledge.com/archives/2017/01/18/who-...
deleted 4y ago
[deleted]
ctvo 4y ago
tl;dr author finds it a hassle to set up proper encryption at rest and rationalizes away why they shouldn’t need to do it. > You used the AWS default key to encrypt a database? Don’t do that? Create a separate account to hold keys. Lock it down like you would your domain DNS or anything else with security implications that will impact your entire company. Share granular read permissions across accounts as needed to encrypt / decrypt. Agreed it’s time consuming to initially setup, but it’s a solved problem and can be implemented with your IaC flavor of choice (CloudFormation directly, CDK, Terraform, …).
- DrRobinson 4y ago
  Hello, author here! What you describe is essentially what I currently do. But I've inherited an infrastructure that was not setup that way, and re-encrypting things has been very time consuming. The company I'm at now use multiple AWS accounts where teams have their own accounts, and it's common for people to forget to use KMS when creating databases or similar. I might have just failed in my search but I couldn't find any way to block default keys via SCPs. If you have any suggestions for that I'm happy to take them!
tekla 4y ago
This is just a "its annoying to do things properly at a bare minimum level" blog post
- meepmorp 4y ago
  I disagree that encryption at rest with AWS is actually the proper way. If someone can get to the actual hardware and steal disks, I can't trust that AWS hasn't also lost control of the encryption keys.
  - cj 4y ago
    Something is better than nothing, though. Just because something isn't 100% perfect in every scenario doesn't mean it shouldn't be done at all. But I agree with your point, if you're really worried about data to the point where you don't trust AWS with encryption keys, you should self-manage your keys and manually encrypt/decrypt data without AWS KMS.
  - ratg13 4y ago
    Swiss Cheese Model https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model AKA defense-in-depth Relying on one control is a recipe for failure, which is why security measures work best when layered. You don’t trust in one control, you trust that you stack enough controls that one of them works.
knorker 4y ago
The author here is missing one big point: The people with physical access in AWS datacenters are not the same people who have access to the encryption keys. In fact, it's likely much more complex than that. The people with software access to the machines very likely don't have access to the key storing system (and definitely not the hardware). This means that the number of people it's possible to bribe to get access to your data is much smaller than if you can just bribe a DC employee to either smuggle the disk out, or make a copy onto a thumb drive. I'm not saying the number of people who could "break glass" and read anyone's data is zero. But it's at least an order of magnitude fewer than the hoards of people employed to swap hard broken drives all day every day. And the people with this "root" access will likely be very well paid, reducing (but not eliminating) risk of bribes.
- dexterdog 4y ago
  You think the datacenter hands at AWS are well-paid? I imagine that's a pretty junior position that is watched with hawk-eye security.
  - Bedon292 4y ago
    Not the datacenter hands. They won't have access to the encryption keys. So the hard drive they do have access to won't be very useful.
  - knorker 4y ago
    Like Bedon292 said, I think you misread my comment. I was saying thanks to encryption they don't have access. Most well paid engineers at AWS won't have access either. Presumably some minimal set would, but they would likely be pretty senior, and well paid. But ideally you'd want one set of people with access to the keys, another with access to the data, and a third with physical access, and no overlap. That way you need three people to conspire. But that's going to be very hard to achieve in practice. But it's not all or nothing. The closer you get to this goal the harder it'll be for someone to not just do it, but do it without triggering a tripwire from the security folks, or at least persist a log entry that if found would get them thrown in prison. Public cloud companies are not like a small startup where everyone has root. (sounds like twitter kinda was, according to recent reports)
Bedon292 4y ago
The primary reason I see for encryption is because all of the disks are shared. Encrypting at rest is to make sure that the next user of the disk is not able to find any of your data. Even if the odds are low, you still don't want any private information leaking on accident. And you have to be able to guarantee it for things like HIPAA or PCI compliance.
- rwmj 4y ago
  Are you saying that EBS exposes previous tenant disk contents when you provision a new disk? I've never heard of that happening. It would be incredibly insecure if true.
  - lazide 4y ago
    It shouldn’t. But do you want to be the example ‘oops’? If FDE is easy to do, it’s usually worth it to reduce the risk to zero.
_8j50 4y ago
It helps you trust the data protection and destruction policies of AWS less. If a disk goes bad, will it be wiped properly before being discarded for example? Or will your confidential data be recoverable by anyone at the landfill? What if an AWS DC tech identifies the bare metal your VM runs on and replaces a disk in the raid and takes the old one home? How good is their security to prevent that from happening? I can think a few ways of getting that past man trap xray scanners and all that physical security.
Ruq 4y ago
If you're complaining about encryption, you're doing it wrong.
- salawat 4y ago
  Counterpoint: How many ways are there to do it wrong than to do it right?
cj 4y ago
> But what problems does it solve? Are you worried about someone breaking into the AWS data center, stealing the specific disks your data is stored on, and restoring and analyzing the disk data to target your organization? Just imagine how much effort such an attack would take. This sounds like it's coming from someone who's forgetting that all of our data in "the cloud" resides in physical buildings throughout the world, which are all high value targets for physical attacks. I don't think it's an antiquated idea to protect against physical data center attacks. It's best practice.
- flaviut 4y ago
  Does AWS not already do full-disk encryption transparently on their side? If they do, I trust them to be good stewards of their keys. If not, then why not? I don't know about the enterprise part of the market, but I know consumer SSDs come with encryption permanently enabled in the firmware.
  - osigurdson 4y ago
    >> but I know consumer SSDs come with encryption permanently enabled in the firmware Where is the encryption key? It seems that it must be in the firmware itself. Presumably it would be possible to find this with enough effort.
    - ckozlowski 4y ago
      The key is stored in KMS, not on the drive firmware. You can read more about how this is done here, in the section at the bottom about "Isolation of Physical Hosts" https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk https://docs.aws.amazon.com/kms/latest/developerguide/concep... But in short, the key is kept in memory on the HSM, and employees don't have access to it. They key can be referenced, but not actually read. It also means that if a user accidentally deletes their key, there's no recovery. That's it. (Pro tip: Deleting a key is a faster mechanism to make data unreadable than deleting the data itself. ;) Disclaimer: I'm an S-TAM with AWS.
    - flaviut 4y ago
greenthrow 4y ago
This is a low quality article. Disk encryption at rest has a bunch of reasons to be required besides someone breaking into AWS to steal the disk (and someone could bribe an employee to do that as easily as bribe an employee to steal your data digitially.) This blog post is like someone complaining "parameterized queries are so much more of a pain than just using string interpolation to build my queries. I'd never allow SQL injection in my app." Hubris of an inexperienced engineer imho.
- DrRobinson 4y ago
  > Hubris of an inexperienced engineer imho. This seems unnecessarily hostile. I've worked with cloud infrastructure and security for more than 10 years. If you have experience of unencrypted disks being stolen from AWS I'm very interested to hear about it. Note, though, that I don't claim one shouldn't encrypt disks, but I consider it to be very low on the list of priorities when it comes to lowering risk. There is almost always risk-lowering actions with better cost:benefit ratio than encrypting disks in the cloud, since that risk is already so low.
slashdev 4y ago
I think the author and the commenters here are using two different threat models. A lot of people here are saying you should encrypt the disks because it’s best practice for security. Or because it protects against some specific, if implausible, scenario. They’re talking about theoretical security. The author is saying the probability that it improves your security is so low that any time it takes away from other security related work is actually making your security worse. That’s a different threat model, and it’s probably the better way to approach security, and the author is probably correct - if he actually uses that saved time to improve security in other ways. I have a question though, what are the laws that require encryption? I can think of HIPPA, SOC2 maybe? What about GDPR?
- lazide 4y ago
  There are secondary effects to not using encryption you can see. For instance, it greatly expands the scope of ‘I lost your data/I exposed your data’ notifications in places like California, and for those under many of those other rulesets. Someone getting access to a repo of encrypted drive images, or someone losing an encrypted drive, doesn’t count. And for reasonableish reasons. It’s a basic risk mitigation/blast radius limiting move. For physical/on-prem especially, since old disks tend to ‘wander’ after retirement, and it’s a great idea to always have had them full disk encrypted to reduce the odds someone sensitive gets exposed years down the line.
- acdha 4y ago
  It’s more that the author is arguing that their inexperience is universally applicable. “Why do banks have guards, I’ve never had someone break into my couch cushions?” Beyond the compliance requirements (e.g. CIS), it’s wrong to assume that disks are always perfectly disposed of – simply never having to worry about featuring in one of those “look what I found on eBay!” posts and notifying your customers is worth the basically non-existent cost of basic encryption. It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally. Beyond that, however, there are two key benefits - the author even almost discovers one of them but didn’t think about it enough: encryption means deletion is probably fast - delete the key and you don’t need to monkey around wiping disks and snapshots. The other, bigger, one is that it can protect against mistakes or compromises. If you share a KMS-encrypted resource accidentally, an attacker won’t have access to the data. An attacker can’t access a resource with a custom KMS policy unless they compromise the specific role which has access or do more invasive things which will trip alerts. Again, not perfect but it protects against common mistakes like the Capitol One firewall breach, which is why all of those standards started recommending it.
whoknew1122 4y ago
> Delete the key and your data is gone This is partially true with respect to EBS (i.e. virtual disks). When you boot up an EC2 instance, the a plaintext data encryption key is loaded into hypervisor memory. As long as the EC2 instance is still running, you can add an additional EBS volume to the instance and then copy data from the disk that has the orphaned key. Even if the underlying KMS key was deleted. The rest of this stuff feels like FUD me. Yes, IAM has identity-based policies and KMS has resource-based policies. And yes, default service KMS keys are unique per account (why would you expect otherwise?). Implementing an encryption program takes forethought. This is true if you're hosting your own data, too. I really don't miss manually rotating drives, fighting with LUKS, and then putting drives in a fireproof safe (which was never locked anyway, so I'm not sure if it would've actually protected anything in an actual fire).
- DrRobinson 4y ago
  > And yes, default service KMS keys are unique per account (why would you expect otherwise?). I expect it to be unique per account, but I would be happy if it was possible to share it with other accounts so one could make cross account backups (it's good practice to have a separate AWS account for backups.) Currently this requires a KMS key, which means data encrypted with the default key must be re-encrypted and that takes a lot of time and effort.
rwmj 4y ago
Right now it's just defence in depth, to protect you if Amazon screws up their physical security. It will make more sense as confidential computing[1] becomes more common. This is because the data can't be accessed by the cloud vendor, assuming the key is generated inside the trusted VM. [1] The trust moves to the CPU vendors instead of the cloud vendors, but if you don't trust CPU vendors then you're going to have a hard time doing anything with computers in the modern world.
- cyberpunk 4y ago
  Is there any tooling you’re aware of in this space? I know of one small group in Europe making something but it’s closed source..
  - rwmj 4y ago
    Attestation is a big mess right now, but there's a consortium working on it, with Red Hat involved: https://www.redhat.com/en/blog/what-confidential-containers-project https://www.redhat.com/en/blog/what-confidential-containers-... https://www.redhat.com/en/blog/understanding-confidential-containers-attestation-flow https://www.redhat.com/en/blog/understanding-confidential-co... (Note confidential containers builds on top of confidential hardware VMs)
hk1337 4y ago
> Delete the key and your data is gone That's pretty much the case for any type of encryption.
osigurdson 4y ago
>> Are you worried about someone breaking into the AWS data center, stealing the specific disks your data is stored on, and restoring and analyzing the disk data to target your organization? Is this really the only vector for data leakage?
- fragmede 4y ago
  How to exfiltrate data is a large topic of discussion. A rogue developer copying the production database to their laptop is another threat, but then they have to copy the data from there, so then you disable USB devices on endpoints. It's an ongoing battle.
gcassie 4y ago
Implicit in this article is the idea that security posturing is a zero-sum game for many companies on the dimensions of both software complexity and time. Adding full disk encryption takes time from other projects and makes the system more complex. That equation needs to pay out. In all likelihood, the reason your data is going to get stolen is a privilege escalation in your app code or a bad actor on your team. Rogue AWS employee swiping your particular hard drive in us-east-1 is way down the list. Full disk encryption does nothing for the first two vectors. I think compliance programs are oriented around pushing companies into complex/expensive system designs thinking that is a proxy for a secure system.
- DrRobinson 4y ago
  You put it really well, I think that's close to how I think about it. Compliance has good sides too though. For example, they force you to think about areas your intuition might otherwise not have gone, so I don't dismiss them but sometimes it makes you spend time on less than optimal things in order to stay compliant.
- fnordpiglet 4y ago
  Aws has done a really good job making encryption fairly simple to enable. It does make some common tasks complex though, like sharing images between accounts. However it’s not fragile or time consuming, and it is typically standardized in an org of any size that requires these sorts of compliance regimes so individual teams don’t need to worry about it. But associating a volume with a key in KMS is not complex or difficult.
  - DrRobinson 4y ago
    I agree. The problem is mainly going from an infrastructure that's not setup like this, to an infrastructure that is. Usually you inherit an infrastructure, and it's usually not set up in this way (in my experience) and then there is a lot of work to re-encrypt the data in order to use KMS rather than the default key. > it is typically standardized in an org I have still not found any SCP I can set that prevent the use of the default key and enforces KMS. If you have one, I'd be happy to take it! If you mean "standardized" as in written on a paper, I'll rely on wishful thinking because people make mistakes or just don't know about it even if it's a standard.
bmcahren 4y ago
The basis of this take is "because it doesn't happen very often, it's snake oil". Has DrRobinson considered that the mere fact they think this is proof it works? The entire point of encryption at rest (on the cloud) is that when any of the following happen you have nothing to worry about. 1. A machine/disk is rendered inoperable and can't be wiped. 2. The data stream coming off of a disk cluster is tapped. 3. An employee steals a disk or is lost. 4. An actor violates their account segmentation and can read raw data from segments of YOUR sectors of the shared disk. 5. SSD firmware goes bad/gets hacked and starts returning incorrect sectors of disk. 6. Memory pointers go bad and return sectors from the wrong area of the disk. It's incredibly naïve to not use encryption at rest on AWS with how incredibly easy and problem free it is to deploy.
- DrRobinson 4y ago
  I agree calling it snake oil is a bit too much, because I know there are benefits. As I mention elsewhere in the thread, I use encryption, but I don't consider it to be of high value compared to other security mitigations one can spend time on. As mentioned in the blog post, setting up encryption isn't always easy and problem free though. Regarding 1, disks are physically destroyed if they are inoperable. Thanks for sharing the list of potential problems, it's always interesting to see how other people think and what they worry about!
  - pixl97 4y ago
    >disks are physically destroyed if they are inoperable 'Supposed to be' is missing from this statement. We have seen many vendors over the years say 'we destroy disks' only to have a disk show up in the wild.
  - bmcahren 4y ago
    I'm equally interested to see what lengths people like yourself will go to when given the option! Where is it you draw the line? Do you bother with Spectre mitigations since Amazon policy is to deny service to those who would attack you? Do you bother requiring authentication on your database since policy is to use a closed VPC? Hell, we haven't seen any man in the middle attacks lately, let's just drop SSL because you know your customers are wired and trust the endpoint networks. SSL is the best example subject to the same common failures (including expiration, key loss, downtime to initially deploy) as you describe disk encryption migrations. Encrypting disks is mandatory and there is zero justification otherwise.
NBJack 4y ago
I'm not sure if the author is aware that they can bring their own key material if they wish to KMS. The cost model is a little different, but you retain complete control of their life cycle across time and regions. It would seem to solve many of their pain points. Additionally, AWS recently added support for a user maintained keystore. As for some of these statements (i.e. loss of key equals loss of data, re-encryption with a new key is expensive to go through an entire disk, etc.), that is exactly the point of encryption and holds true beyond the cloud. It is not meant to be trivial; it is a layer of security that trades convenience and often some level of performance.
hanselot 4y ago
What about protecting your data from AWS?
based2 4y ago
https://docs.aws.amazon.com/workspaces/latest/adminguide/encrypt-workspaces.html#encryption_limits https://docs.aws.amazon.com/workspaces/latest/adminguide/enc...
Eleison23 4y ago
commandersaki 4y ago
I agree with the author that it's worthwhile spending your time elsewhere than adding encryption. I think the real issue is higher-ups think that if data is encrypted that somehow mitigates meaningful exfiltration, but just look at data breaches like Capital One and you'll see that's not the case. The whole stealing the (correct) hard drive or concerning yourself with a host-level attack should be at the very bottom of the list as far as your threat model goes -- these are not really typical of the surface area of most deployments and you're better off focusing on the surface area such as application security, transport security, having your applications perform the encryption than relying on AWS controls, etc.
MattPalmer1086 4y ago
Imagine if disk encryption were not routinely used. The disks of the cloud providers would be a goldmine of sensitive corporate data and the incentive for them to go missing correspondingly higher.

44 points by DrRobinson 4y ago

andorov 4y ago

What about the disk being reused by another process without being thoroughly wiped by AWS?

colonelpopcorn 4y ago
That's an interesting idea. I imagine they don't do DoD wipes of their disks because they are probably multi-tenant, right? Can a guest operating system do forensic analysis on the host disk?
- britneybitch 4y ago
  AWS wipes disks before reusing them (loosely speaking--an EBS volume doesn't correspond 1:1 to a physical disk). Forensic analysis of overwritten data requires disassembling the physical disk, so no, it's not possible.
whoknew1122 4y ago
Disks are wiped immediately before they are made available as a new volume. If your policies require a certain level of wipe, you should wipe the disk before releasing it.

likecarter 4y ago

Stealing disks does happen, especially on-premise, which is where these policies originated. Also, sometimes there are mistakes with decommissioning old drives, and you wouldn’t want your data discovered in a landfill somewhere.

eloff 4y ago
The author is talking specifically about AWS. The odds that there is a mistake decommissioning the disk that leaves the data intact, times that somebody salvaged it from a landfill, times that they care about your data is basically zero. Which means a logical person should worry about everything else.
- im3w1l 4y ago
  What are the odds that some arranges for all those things to happen though? When you try to go after them they will have plausible deniability.
  - eloff 4y ago
    That’s an interesting attack vector. Bribe someone to replace the disk without wiping it, and be in a position to intercept it after that.
    - DrRobinson 4y ago
      This would require you or the person in the data center to know which customer is using which disk. And data isn't stored on just one disk, it's spread out over multiple disks and many customers have shards of data stored on the same disk. So even if this did happen, the would only get fragments of data.
  - fnordpiglet 4y ago
    Disk management and destruction is largely automated. That’s extraordinarily unlikely.
- knorker 4y ago
  You forgot to multiply by the millions of disks they presumably go through every year.
  - eloff 4y ago
    You don’t though. Your risk calculation is the same.
rolenthedeep 4y ago
I once bought a "new" hard drive off Amazon. The connectors looked suspiciously scratched up, like it had been used before. When I dug deeper, they had wiped the SMART data and the partition table, but it was absolutely full of readable data. I found clear text server logs indicating that this drive was in a backblaze center for several thousand hours.
- dexterdog 4y ago
  Did you report that to backblaze? I would think a bug bounty would be paid on something like that.
  - Jensson 4y ago
    I'd worry they would sue for hacking to cover it up, dumber things has happened to nice developers who report vulnerabilities.
    - brianwski 4y ago
      Disclaimer: I work at Backblaze, and I was here first. > I'd worry they would sue for... If you are referring to Backblaze, we're not going to "sue" anybody for anything. We (Backblaze) have dealt with a bunch of frivolous lawsuits (and patent trolls) over the years suing us, and OMG we're not going to instigate any lawsuits over some honest person legitimately reporting some issue and being helpful. It isn't going to happen. Our reputation is important to us. Not just for Backblaze: I'm saying the individuals that founded Backblaze and those people that work now here base our entire existence and careers and the number one marketing efforts at Backblaze are based around we are trying to be "the good guys" and transparent and acting like it. There is no possibly world where we try to suppress a screwup like this through legal means. That would be a PR debacle of epic proportions. If something went wrong, let's shine a spotlight on that cockroach and figure it out together. I'm not sure the exact drive we are all talking about, but my first guess would be a customer ordered a $189 "USB Restore" all their data shipped to them on an encrypted drive) and we (Backblaze) shipped the customer a USB restore drive and they are subsequently selling it (after copying their restore off of it) on the open market. If it is above 8 TBytes this is absolutely *NOT* the case and we should get to the bottom of it. Without lawyers mucking up the situation.
kube-system 4y ago
They don't go into a landfill unless they're broken. Old drives that someone could still get the data off of end up on eBay.
lokar 4y ago
Google has very tight physical security (metal detectors, etc) and disks are either confirmed erased or shredded I assume aws is similar
- LammyL 4y ago
  Google mostly runs their own data centers. Amazon mostly rents space in commercial data centers with lots of different companies. Aws is probably pretty secure but likely less physically secure compared to Google if you get into the nitty gritty details.
  - fnordpiglet 4y ago
    This is false. Aws builds and operates their own data centers. The design and architecture of their data centers are highly standardized and an enormous amount of the build is fully automated. When they do lease they lease just a basic power and fiber build and they build their data center ontop of the base. Given how much custom equipment they run at the data center level it would be impractical to do anything that was collocated. But most data centers for aws, especially in Virginia, are on Amazon owned land. As far as I am aware the only collocations are in the satellite zones, outposts, and in peering locations. https://www.datacenterknowledge.com/archives/2017/01/18/who-leased-the-most-data-center-space-in-2016 https://www.datacenterknowledge.com/archives/2017/01/18/who-...

deleted 4y ago

[deleted]

ctvo 4y ago

tl;dr author finds it a hassle to set up proper encryption at rest and rationalizes away why they shouldn’t need to do it. > You used the AWS default key to encrypt a database? Don’t do that? Create a separate account to hold keys. Lock it down like you would your domain DNS or anything else with security implications that will impact your entire company. Share granular read permissions across accounts as needed to encrypt / decrypt. Agreed it’s time consuming to initially setup, but it’s a solved problem and can be implemented with your IaC flavor of choice (CloudFormation directly, CDK, Terraform, …).

DrRobinson 4y ago
Hello, author here! What you describe is essentially what I currently do. But I've inherited an infrastructure that was not setup that way, and re-encrypting things has been very time consuming. The company I'm at now use multiple AWS accounts where teams have their own accounts, and it's common for people to forget to use KMS when creating databases or similar. I might have just failed in my search but I couldn't find any way to block default keys via SCPs. If you have any suggestions for that I'm happy to take them!

tekla 4y ago

This is just a "its annoying to do things properly at a bare minimum level" blog post

meepmorp 4y ago
I disagree that encryption at rest with AWS is actually the proper way. If someone can get to the actual hardware and steal disks, I can't trust that AWS hasn't also lost control of the encryption keys.
- cj 4y ago
  Something is better than nothing, though. Just because something isn't 100% perfect in every scenario doesn't mean it shouldn't be done at all. But I agree with your point, if you're really worried about data to the point where you don't trust AWS with encryption keys, you should self-manage your keys and manually encrypt/decrypt data without AWS KMS.
- ratg13 4y ago
  Swiss Cheese Model https://en.wikipedia.org/wiki/Swiss_cheese_model https://en.wikipedia.org/wiki/Swiss_cheese_model AKA defense-in-depth Relying on one control is a recipe for failure, which is why security measures work best when layered. You don’t trust in one control, you trust that you stack enough controls that one of them works.

knorker 4y ago

The author here is missing one big point: The people with physical access in AWS datacenters are not the same people who have access to the encryption keys. In fact, it's likely much more complex than that. The people with software access to the machines very likely don't have access to the key storing system (and definitely not the hardware). This means that the number of people it's possible to bribe to get access to your data is much smaller than if you can just bribe a DC employee to either smuggle the disk out, or make a copy onto a thumb drive. I'm not saying the number of people who could "break glass" and read anyone's data is zero. But it's at least an order of magnitude fewer than the hoards of people employed to swap hard broken drives all day every day. And the people with this "root" access will likely be very well paid, reducing (but not eliminating) risk of bribes.

dexterdog 4y ago
You think the datacenter hands at AWS are well-paid? I imagine that's a pretty junior position that is watched with hawk-eye security.
- Bedon292 4y ago
  Not the datacenter hands. They won't have access to the encryption keys. So the hard drive they do have access to won't be very useful.
- knorker 4y ago
  Like Bedon292 said, I think you misread my comment. I was saying thanks to encryption they don't have access. Most well paid engineers at AWS won't have access either. Presumably some minimal set would, but they would likely be pretty senior, and well paid. But ideally you'd want one set of people with access to the keys, another with access to the data, and a third with physical access, and no overlap. That way you need three people to conspire. But that's going to be very hard to achieve in practice. But it's not all or nothing. The closer you get to this goal the harder it'll be for someone to not just do it, but do it without triggering a tripwire from the security folks, or at least persist a log entry that if found would get them thrown in prison. Public cloud companies are not like a small startup where everyone has root. (sounds like twitter kinda was, according to recent reports)

Bedon292 4y ago

The primary reason I see for encryption is because all of the disks are shared. Encrypting at rest is to make sure that the next user of the disk is not able to find any of your data. Even if the odds are low, you still don't want any private information leaking on accident. And you have to be able to guarantee it for things like HIPAA or PCI compliance.

rwmj 4y ago
Are you saying that EBS exposes previous tenant disk contents when you provision a new disk? I've never heard of that happening. It would be incredibly insecure if true.
- lazide 4y ago
  It shouldn’t. But do you want to be the example ‘oops’? If FDE is easy to do, it’s usually worth it to reduce the risk to zero.

_8j50 4y ago

It helps you trust the data protection and destruction policies of AWS less. If a disk goes bad, will it be wiped properly before being discarded for example? Or will your confidential data be recoverable by anyone at the landfill? What if an AWS DC tech identifies the bare metal your VM runs on and replaces a disk in the raid and takes the old one home? How good is their security to prevent that from happening? I can think a few ways of getting that past man trap xray scanners and all that physical security.

Ruq 4y ago

If you're complaining about encryption, you're doing it wrong.

salawat 4y ago
Counterpoint: How many ways are there to do it wrong than to do it right?

cj 4y ago

> But what problems does it solve? Are you worried about someone breaking into the AWS data center, stealing the specific disks your data is stored on, and restoring and analyzing the disk data to target your organization? Just imagine how much effort such an attack would take. This sounds like it's coming from someone who's forgetting that all of our data in "the cloud" resides in physical buildings throughout the world, which are all high value targets for physical attacks. I don't think it's an antiquated idea to protect against physical data center attacks. It's best practice.

flaviut 4y ago
Does AWS not already do full-disk encryption transparently on their side? If they do, I trust them to be good stewards of their keys. If not, then why not? I don't know about the enterprise part of the market, but I know consumer SSDs come with encryption permanently enabled in the firmware.
- osigurdson 4y ago
  >> but I know consumer SSDs come with encryption permanently enabled in the firmware Where is the encryption key? It seems that it must be in the firmware itself. Presumably it would be possible to find this with enough effort.
  - ckozlowski 4y ago
    The key is stored in KMS, not on the drive firmware. You can read more about how this is done here, in the section at the bottom about "Isolation of Physical Hosts" https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk https://docs.aws.amazon.com/kms/latest/developerguide/concep... But in short, the key is kept in memory on the HSM, and employees don't have access to it. They key can be referenced, but not actually read. It also means that if a user accidentally deletes their key, there's no recovery. That's it. (Pro tip: Deleting a key is a faster mechanism to make data unreadable than deleting the data itself. ;) Disclaimer: I'm an S-TAM with AWS.
  - flaviut 4y ago

greenthrow 4y ago

This is a low quality article. Disk encryption at rest has a bunch of reasons to be required besides someone breaking into AWS to steal the disk (and someone could bribe an employee to do that as easily as bribe an employee to steal your data digitially.) This blog post is like someone complaining "parameterized queries are so much more of a pain than just using string interpolation to build my queries. I'd never allow SQL injection in my app." Hubris of an inexperienced engineer imho.

DrRobinson 4y ago
> Hubris of an inexperienced engineer imho. This seems unnecessarily hostile. I've worked with cloud infrastructure and security for more than 10 years. If you have experience of unencrypted disks being stolen from AWS I'm very interested to hear about it. Note, though, that I don't claim one shouldn't encrypt disks, but I consider it to be very low on the list of priorities when it comes to lowering risk. There is almost always risk-lowering actions with better cost:benefit ratio than encrypting disks in the cloud, since that risk is already so low.

slashdev 4y ago

I think the author and the commenters here are using two different threat models. A lot of people here are saying you should encrypt the disks because it’s best practice for security. Or because it protects against some specific, if implausible, scenario. They’re talking about theoretical security. The author is saying the probability that it improves your security is so low that any time it takes away from other security related work is actually making your security worse. That’s a different threat model, and it’s probably the better way to approach security, and the author is probably correct - if he actually uses that saved time to improve security in other ways. I have a question though, what are the laws that require encryption? I can think of HIPPA, SOC2 maybe? What about GDPR?

lazide 4y ago
There are secondary effects to not using encryption you can see. For instance, it greatly expands the scope of ‘I lost your data/I exposed your data’ notifications in places like California, and for those under many of those other rulesets. Someone getting access to a repo of encrypted drive images, or someone losing an encrypted drive, doesn’t count. And for reasonableish reasons. It’s a basic risk mitigation/blast radius limiting move. For physical/on-prem especially, since old disks tend to ‘wander’ after retirement, and it’s a great idea to always have had them full disk encrypted to reduce the odds someone sensitive gets exposed years down the line.
acdha 4y ago
It’s more that the author is arguing that their inexperience is universally applicable. “Why do banks have guards, I’ve never had someone break into my couch cushions?” Beyond the compliance requirements (e.g. CIS), it’s wrong to assume that disks are always perfectly disposed of – simply never having to worry about featuring in one of those “look what I found on eBay!” posts and notifying your customers is worth the basically non-existent cost of basic encryption. It’s something you turn on and basically never think about again - the performance hit disappeared around 2010 and it’s easy to enable globally. Beyond that, however, there are two key benefits - the author even almost discovers one of them but didn’t think about it enough: encryption means deletion is probably fast - delete the key and you don’t need to monkey around wiping disks and snapshots. The other, bigger, one is that it can protect against mistakes or compromises. If you share a KMS-encrypted resource accidentally, an attacker won’t have access to the data. An attacker can’t access a resource with a custom KMS policy unless they compromise the specific role which has access or do more invasive things which will trip alerts. Again, not perfect but it protects against common mistakes like the Capitol One firewall breach, which is why all of those standards started recommending it.

whoknew1122 4y ago

> Delete the key and your data is gone This is partially true with respect to EBS (i.e. virtual disks). When you boot up an EC2 instance, the a plaintext data encryption key is loaded into hypervisor memory. As long as the EC2 instance is still running, you can add an additional EBS volume to the instance and then copy data from the disk that has the orphaned key. Even if the underlying KMS key was deleted. The rest of this stuff feels like FUD me. Yes, IAM has identity-based policies and KMS has resource-based policies. And yes, default service KMS keys are unique per account (why would you expect otherwise?). Implementing an encryption program takes forethought. This is true if you're hosting your own data, too. I really don't miss manually rotating drives, fighting with LUKS, and then putting drives in a fireproof safe (which was never locked anyway, so I'm not sure if it would've actually protected anything in an actual fire).

DrRobinson 4y ago
> And yes, default service KMS keys are unique per account (why would you expect otherwise?). I expect it to be unique per account, but I would be happy if it was possible to share it with other accounts so one could make cross account backups (it's good practice to have a separate AWS account for backups.) Currently this requires a KMS key, which means data encrypted with the default key must be re-encrypted and that takes a lot of time and effort.

rwmj 4y ago

Right now it's just defence in depth, to protect you if Amazon screws up their physical security. It will make more sense as confidential computing[1] becomes more common. This is because the data can't be accessed by the cloud vendor, assuming the key is generated inside the trusted VM. [1] The trust moves to the CPU vendors instead of the cloud vendors, but if you don't trust CPU vendors then you're going to have a hard time doing anything with computers in the modern world.

cyberpunk 4y ago
Is there any tooling you’re aware of in this space? I know of one small group in Europe making something but it’s closed source..
- rwmj 4y ago
  Attestation is a big mess right now, but there's a consortium working on it, with Red Hat involved: https://www.redhat.com/en/blog/what-confidential-containers-project https://www.redhat.com/en/blog/what-confidential-containers-... https://www.redhat.com/en/blog/understanding-confidential-containers-attestation-flow https://www.redhat.com/en/blog/understanding-confidential-co... (Note confidential containers builds on top of confidential hardware VMs)

hk1337 4y ago

> Delete the key and your data is gone That's pretty much the case for any type of encryption.

osigurdson 4y ago

>> Are you worried about someone breaking into the AWS data center, stealing the specific disks your data is stored on, and restoring and analyzing the disk data to target your organization? Is this really the only vector for data leakage?

fragmede 4y ago
How to exfiltrate data is a large topic of discussion. A rogue developer copying the production database to their laptop is another threat, but then they have to copy the data from there, so then you disable USB devices on endpoints. It's an ongoing battle.

gcassie 4y ago

Implicit in this article is the idea that security posturing is a zero-sum game for many companies on the dimensions of both software complexity and time. Adding full disk encryption takes time from other projects and makes the system more complex. That equation needs to pay out. In all likelihood, the reason your data is going to get stolen is a privilege escalation in your app code or a bad actor on your team. Rogue AWS employee swiping your particular hard drive in us-east-1 is way down the list. Full disk encryption does nothing for the first two vectors. I think compliance programs are oriented around pushing companies into complex/expensive system designs thinking that is a proxy for a secure system.

DrRobinson 4y ago
You put it really well, I think that's close to how I think about it. Compliance has good sides too though. For example, they force you to think about areas your intuition might otherwise not have gone, so I don't dismiss them but sometimes it makes you spend time on less than optimal things in order to stay compliant.
fnordpiglet 4y ago
Aws has done a really good job making encryption fairly simple to enable. It does make some common tasks complex though, like sharing images between accounts. However it’s not fragile or time consuming, and it is typically standardized in an org of any size that requires these sorts of compliance regimes so individual teams don’t need to worry about it. But associating a volume with a key in KMS is not complex or difficult.
- DrRobinson 4y ago
  I agree. The problem is mainly going from an infrastructure that's not setup like this, to an infrastructure that is. Usually you inherit an infrastructure, and it's usually not set up in this way (in my experience) and then there is a lot of work to re-encrypt the data in order to use KMS rather than the default key. > it is typically standardized in an org I have still not found any SCP I can set that prevent the use of the default key and enforces KMS. If you have one, I'd be happy to take it! If you mean "standardized" as in written on a paper, I'll rely on wishful thinking because people make mistakes or just don't know about it even if it's a standard.

bmcahren 4y ago

The basis of this take is "because it doesn't happen very often, it's snake oil". Has DrRobinson considered that the mere fact they think this is proof it works? The entire point of encryption at rest (on the cloud) is that when any of the following happen you have nothing to worry about. 1. A machine/disk is rendered inoperable and can't be wiped. 2. The data stream coming off of a disk cluster is tapped. 3. An employee steals a disk or is lost. 4. An actor violates their account segmentation and can read raw data from segments of YOUR sectors of the shared disk. 5. SSD firmware goes bad/gets hacked and starts returning incorrect sectors of disk. 6. Memory pointers go bad and return sectors from the wrong area of the disk. It's incredibly naïve to not use encryption at rest on AWS with how incredibly easy and problem free it is to deploy.

DrRobinson 4y ago
I agree calling it snake oil is a bit too much, because I know there are benefits. As I mention elsewhere in the thread, I use encryption, but I don't consider it to be of high value compared to other security mitigations one can spend time on. As mentioned in the blog post, setting up encryption isn't always easy and problem free though. Regarding 1, disks are physically destroyed if they are inoperable. Thanks for sharing the list of potential problems, it's always interesting to see how other people think and what they worry about!
- pixl97 4y ago
  >disks are physically destroyed if they are inoperable 'Supposed to be' is missing from this statement. We have seen many vendors over the years say 'we destroy disks' only to have a disk show up in the wild.
- bmcahren 4y ago
  I'm equally interested to see what lengths people like yourself will go to when given the option! Where is it you draw the line? Do you bother with Spectre mitigations since Amazon policy is to deny service to those who would attack you? Do you bother requiring authentication on your database since policy is to use a closed VPC? Hell, we haven't seen any man in the middle attacks lately, let's just drop SSL because you know your customers are wired and trust the endpoint networks. SSL is the best example subject to the same common failures (including expiration, key loss, downtime to initially deploy) as you describe disk encryption migrations. Encrypting disks is mandatory and there is zero justification otherwise.

NBJack 4y ago

I'm not sure if the author is aware that they can bring their own key material if they wish to KMS. The cost model is a little different, but you retain complete control of their life cycle across time and regions. It would seem to solve many of their pain points. Additionally, AWS recently added support for a user maintained keystore. As for some of these statements (i.e. loss of key equals loss of data, re-encryption with a new key is expensive to go through an entire disk, etc.), that is exactly the point of encryption and holds true beyond the cloud. It is not meant to be trivial; it is a layer of security that trades convenience and often some level of performance.

hanselot 4y ago

What about protecting your data from AWS?

based2 4y ago

https://docs.aws.amazon.com/workspaces/latest/adminguide/encrypt-workspaces.html#encryption_limits https://docs.aws.amazon.com/workspaces/latest/adminguide/enc...

Eleison23 4y ago

commandersaki 4y ago

I agree with the author that it's worthwhile spending your time elsewhere than adding encryption. I think the real issue is higher-ups think that if data is encrypted that somehow mitigates meaningful exfiltration, but just look at data breaches like Capital One and you'll see that's not the case. The whole stealing the (correct) hard drive or concerning yourself with a host-level attack should be at the very bottom of the list as far as your threat model goes -- these are not really typical of the surface area of most deployments and you're better off focusing on the surface area such as application security, transport security, having your applications perform the encryption than relying on AWS controls, etc.

MattPalmer1086 4y ago

Imagine if disk encryption were not routinely used. The disks of the cloud providers would be a goldmine of sensitive corporate data and the incentive for them to go missing correspondingly higher.