8 ms·

Encrypting Postgres Data at Rest in Kubernetes

108 points by plaur782 5y ago

speedgoose 5y ago
These automatically encrypted disks with managed keys in the clouds are nice to check the "encrypted at rest" checkbox for security audits but I think that they add little security. In most scenarios I can think of, both the data and the keys will be accessible to the attacker. And the ones where the attacker would have access to only the encrypted data seems very unlikely, like physical access to the data center with the knowledge of where is physically stored the data. But I would be gladly proven wrong.
- slifin 5y ago
  So many developers think encryption means it's completely fine to have unrelated security failures Encryption at rest is a protection against physical theft that's it Encrypting a hot relational database is madness but I've seen several bad attempts at it anyway
- CamouflagedKiwi 5y ago
  Agreed, although it does mean that if you release the volume you aren't relying on the cloud provider to wipe it before reuse (AFAIK they generally do that, AWS definitely does but it's possible other providers might not, especially if you're on some more custom OpenShift style setup).
  - jeffbee 5y ago
    GCE persistent disks are always encrypted anyway, with a customer-provided key if the customer chooses, or an automatic ephemeral key otherwise. I think the only storage option in GCE that doesn't feature customer encryption keys is locally-attached SSD, where you'd need to layer your own crypto on top. Definitely don't rely on cloud providers to wipe your data.
- latch 5y ago
  In most cases, you're trusting other organizations to dispose of the disk, ensure they can be securely re-used (1) and that their employees will generally behave correctly and ethically. But still, in general, I agree. I fill in these security questionnaires maybe once every couple months, and I'm starting to see a clear shift from encryption of data-at-rest to encryption of data-in-use. Which mostly just feel like an even more tedious form of security theater. (1) https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097
- exitheone 5y ago
  But that is exactly the point. You want to ensure that disks later sold by your cloud provider do not contain your data in plain. That's the whole point with encryption at rest. It's not supposed to protect you against an attacker with root access to your server.
  - deleted 5y ago
    [deleted]
  - jeffbee 5y ago
    But encryption at rest does protect you from insider or attacker access with full privileges on the servers where the data is stored. The data would necessarily be visible to a root user on the system where the data is being decrypted by an application, but that's not necessarily the same node.
  - deleted 5y ago
    [deleted]
  - yashap 5y ago
    A good, fairly recent example from the city I live in, of hard drives being sold that were (accidentally) absolutely chock full of unencrypted customer PII: https://www.google.com/amp/s/globalnews.ca/news/4476625/ncix-server-data-breach/amp/ https://www.google.com/amp/s/globalnews.ca/news/4476625/ncix...
  - itisit 5y ago
    The big cloud providers destroy old drives.
- zorked 5y ago
  Not everything is physical access though. Suppose someone comes up with a way to force S3 to read random storage blocks, or to make a virtualized storage device read past its boundaries in the underlying storage, or to intercept another VM's ring buffer in the hypervisor. It's an entirely different scenario to read plaintext vs read something encrypted. Even if all it protects is against the scenario of the cloud provider forgetting to wipe a disk, that's worth it.
  - speedgoose 5y ago
    That's a good point. I prefer to encrypt my files before they go to S3, it's easy to do in applications or using Minio as a gateway.
    - dividedbyzero 5y ago
      Minio can do that transparently?
      - speedgoose 5y ago
        I think so, but I'm not sure about their long term plans about it.
- ATsch 5y ago
  I agree in the general case but I think there are uses for it that only go slightly beyond the fully automatic version. For example selectively encrypting less frequently needed sensitive fields can give you more granular access control and logging, especially if you have to check in with another service to perform the decryption instead of having the key available locally.
- throwaway984393 5y ago
  > And the ones where the attacker would have access to only the encrypted data seems very unlikely, like physical access to the data center This is the primary purpose of encryption at rest. It is not unlikely. Have you ever worked in an average datacenter? Some of them have very little physical security and don't monitor employees' access to cages. And if you have a cage on the same floor as another customer, all it takes is a fake name badge and a clipboard to walk up to someone with their cage open and say you're doing a routine inspection. Walk into the cage, pop out a drive, put it under your clipboard, walk away. Ask physical security pentesters how "difficult" it is to steal from a datacenter. And let's not even get into dumpster diving, where clients regularly toss entire servers without erasing disks into dumpsters. Like someone else said, it also is a good practice in multi-tenant configurations where a misconfigured storage or compute backplane could expose data to the wrong tenant.
  - hsbauauvhabzb 5y ago
    Any case studies related to AWS or azure?
- bob1029 5y ago
  > But I would be gladly proven wrong. All I can do is support your position. Our customers are interested in the idea of encryption-at-rest of any sensitive business data that we have on disk at any time. I work in banking, so I am ethically bound to provide the most brutally-honest takes regarding security models to our customers. When developing proposals for encryption-at-rest we have to cast everything in the worst possible light. We started with looking at things like DPAPI but that is almost a joke for our use cases. We are more worried about something along the lines of - A senior IT administrator has access to all application servers with full admin/root and can trivially dump customer databases to the dark net for crypto. When modeling for this scenario, we have to consider both physical and digital controls. Digital controls alone will never provide a comprehensive solution for this kind of adversary. Our approach looks something like this: The customer has to install HSM(s) in their environment. This would typically be something living in a nested secure network or USB dongle directly attached to each server. These secure environments/items must not ever be physically accessible unless at least 2 employees are present at the same time. Each of our application servers which transact PII must be configured to talk to the HSMs. Multiple HSMs provide redundancy in a production environment, mitigating downtime risk. The cryptoscheme is a key-wrapping approach, where working keys are managed per user work item. When a work item is requested for the first time, the HSM has to unwrap its working key (which is stored encrypted on the application server). There is a bit of a mild tradeoff between security and performance here. Hypothetically, we could require the HSM encrypt the payload itself on every I/O, but for our product that would quickly saturate the capacity of even the most expensive HSMs. The policy is that an untouched work item has its key zeroed out in memory after X seconds. Working keys that timed out of memory are marked for rekey on next retrieval. We have also modeled for remote debugger scenarios, and believe that simple firewall and other access controls can mitigate most of those vectors. This is very much a defense-in-depth strategy, so I am only presenting a small piece of that puzzle here.
- tw04 5y ago
  You're kind of missing the point. The whole point of data at rest encryption is: if this drive dies and gets thrown into a dumpster, I don't want some random shmuck to pull it out, put it in a server, run a recovery program against it, and get my customer's SSNs. Now, I would hope Amazon shreds drives that are declared dead, but I wouldn't want to risk my business on it. It has almost nothing to do with protection against a targeted attack and everything to do with chain of custody.
CamouflagedKiwi 5y ago
Was hoping for something a little more profound than "use an encrypted storageclass for your volumes".
- Fiahil 5y ago
  Yes, what they explained is entirely based on encrypted EBS volume. Nothing related with Kubernetes apart from the ˋencrypted: true` in the StorageClass.
sylr 5y ago
Am I the only one enforcing a strict no database in kubernetes policy ?
- plaur782 5y ago
  No. You are not alone. There are still a number of organizations that are cautious about deploying databases in Kubernetes. That said, various third party surveys as well as anecdotal evidence of what we are seeing at Crunchy Data suggests that deploying databases on Kubernetes is increasingly common. The degree to which it makes sense often depends on whether or not the organization is standardizing around Kubernetes for their deployment model more generally.
- zinclozenge 5y ago
  You're not the only one, obviously. But that stance is not really relevant anymore. Just about every database as a service company will run their databases in kubernetes now.
- Sebb767 5y ago
  Honest question: Why is that? What is wrong with your DB running in Kubernetes and what do you suggest as alternative?
  - sylr 5y ago
    I consider anything in kubernetes disposable. I want to be able to lift and shift anything in a new cluster without any migration in a matter of minutes. Anything "stateful" like a database breaks this paradigm. I have nothing against databases used as cache that can be "re-filled" upon re-creation, but I believe anything holding business critical data shall be held outside of a kubernetes cluster. Why, because being one command away deleting your StatefulSet, Helm Release ... etc scares the shit out of me. You can of course minimize the risk with correct RBAC, ensure proper backup/restore migrations but that require lots of staff and efforts I can't spare. So until I can be reassured that I have all the tooling that can recover rapidly any catastrophic failure/mishap, and that all this tooling is tested monthly, I enforce using managed databases services.
handrous 5y ago
Tangentially related: what's state-of-the-art for data protection & access control for small organizations? One runs into the "someone's gotta be trusted with the master keys" problem there so early & often that all the "big" solutions feel silly. Do small shops just farm this out via SaaS and hope their provider's doing the right thing? (the answer back in the day, and perhaps still, was just "they don't really worry about it at all, and hope nothing goes wrong")
ianlevesque 5y ago
This post was really laying it on a bit thick on the marketing, with three mentions of their own products before even finishing the introduction. I know that’s the point of most of these posts but then when the content was also a product, it’s too much. Pass.

108 points by plaur782 5y ago

These automatically encrypted disks with managed keys in the clouds are nice to check the "encrypted at rest" checkbox for security audits but I think that they add little security. In most scenarios I can think of, both the data and the keys will be accessible to the attacker. And the ones where the attacker would have access to only the encrypted data seems very unlikely, like physical access to the data center with the knowledge of where is physically stored the data. But I would be gladly proven wrong.

slifin 5y ago
So many developers think encryption means it's completely fine to have unrelated security failures Encryption at rest is a protection against physical theft that's it Encrypting a hot relational database is madness but I've seen several bad attempts at it anyway
CamouflagedKiwi 5y ago
Agreed, although it does mean that if you release the volume you aren't relying on the cloud provider to wipe it before reuse (AFAIK they generally do that, AWS definitely does but it's possible other providers might not, especially if you're on some more custom OpenShift style setup).
- jeffbee 5y ago
  GCE persistent disks are always encrypted anyway, with a customer-provided key if the customer chooses, or an automatic ephemeral key otherwise. I think the only storage option in GCE that doesn't feature customer encryption keys is locally-attached SSD, where you'd need to layer your own crypto on top. Definitely don't rely on cloud providers to wipe your data.
latch 5y ago
In most cases, you're trusting other organizations to dispose of the disk, ensure they can be securely re-used (1) and that their employees will generally behave correctly and ethically. But still, in general, I agree. I fill in these security questionnaires maybe once every couple months, and I'm starting to see a clear shift from encryption of data-at-rest to encryption of data-in-use. Which mostly just feel like an even more tedious form of security theater. (1) https://news.ycombinator.com/item?id=6983097 https://news.ycombinator.com/item?id=6983097
exitheone 5y ago
But that is exactly the point. You want to ensure that disks later sold by your cloud provider do not contain your data in plain. That's the whole point with encryption at rest. It's not supposed to protect you against an attacker with root access to your server.
- deleted 5y ago
  [deleted]
- jeffbee 5y ago
  But encryption at rest does protect you from insider or attacker access with full privileges on the servers where the data is stored. The data would necessarily be visible to a root user on the system where the data is being decrypted by an application, but that's not necessarily the same node.
- deleted 5y ago
  [deleted]
- yashap 5y ago
  A good, fairly recent example from the city I live in, of hard drives being sold that were (accidentally) absolutely chock full of unencrypted customer PII: https://www.google.com/amp/s/globalnews.ca/news/4476625/ncix-server-data-breach/amp/ https://www.google.com/amp/s/globalnews.ca/news/4476625/ncix...
- itisit 5y ago
  The big cloud providers destroy old drives.
zorked 5y ago
Not everything is physical access though. Suppose someone comes up with a way to force S3 to read random storage blocks, or to make a virtualized storage device read past its boundaries in the underlying storage, or to intercept another VM's ring buffer in the hypervisor. It's an entirely different scenario to read plaintext vs read something encrypted. Even if all it protects is against the scenario of the cloud provider forgetting to wipe a disk, that's worth it.
- speedgoose 5y ago
  That's a good point. I prefer to encrypt my files before they go to S3, it's easy to do in applications or using Minio as a gateway.
  - dividedbyzero 5y ago
    Minio can do that transparently?
    - speedgoose 5y ago
      I think so, but I'm not sure about their long term plans about it.
ATsch 5y ago
I agree in the general case but I think there are uses for it that only go slightly beyond the fully automatic version. For example selectively encrypting less frequently needed sensitive fields can give you more granular access control and logging, especially if you have to check in with another service to perform the decryption instead of having the key available locally.
throwaway984393 5y ago
> And the ones where the attacker would have access to only the encrypted data seems very unlikely, like physical access to the data center This is the primary purpose of encryption at rest. It is not unlikely. Have you ever worked in an average datacenter? Some of them have very little physical security and don't monitor employees' access to cages. And if you have a cage on the same floor as another customer, all it takes is a fake name badge and a clipboard to walk up to someone with their cage open and say you're doing a routine inspection. Walk into the cage, pop out a drive, put it under your clipboard, walk away. Ask physical security pentesters how "difficult" it is to steal from a datacenter. And let's not even get into dumpster diving, where clients regularly toss entire servers without erasing disks into dumpsters. Like someone else said, it also is a good practice in multi-tenant configurations where a misconfigured storage or compute backplane could expose data to the wrong tenant.
- hsbauauvhabzb 5y ago
  Any case studies related to AWS or azure?
bob1029 5y ago
> But I would be gladly proven wrong. All I can do is support your position. Our customers are interested in the idea of encryption-at-rest of any sensitive business data that we have on disk at any time. I work in banking, so I am ethically bound to provide the most brutally-honest takes regarding security models to our customers. When developing proposals for encryption-at-rest we have to cast everything in the worst possible light. We started with looking at things like DPAPI but that is almost a joke for our use cases. We are more worried about something along the lines of - A senior IT administrator has access to all application servers with full admin/root and can trivially dump customer databases to the dark net for crypto. When modeling for this scenario, we have to consider both physical and digital controls. Digital controls alone will never provide a comprehensive solution for this kind of adversary. Our approach looks something like this: The customer has to install HSM(s) in their environment. This would typically be something living in a nested secure network or USB dongle directly attached to each server. These secure environments/items must not ever be physically accessible unless at least 2 employees are present at the same time. Each of our application servers which transact PII must be configured to talk to the HSMs. Multiple HSMs provide redundancy in a production environment, mitigating downtime risk. The cryptoscheme is a key-wrapping approach, where working keys are managed per user work item. When a work item is requested for the first time, the HSM has to unwrap its working key (which is stored encrypted on the application server). There is a bit of a mild tradeoff between security and performance here. Hypothetically, we could require the HSM encrypt the payload itself on every I/O, but for our product that would quickly saturate the capacity of even the most expensive HSMs. The policy is that an untouched work item has its key zeroed out in memory after X seconds. Working keys that timed out of memory are marked for rekey on next retrieval. We have also modeled for remote debugger scenarios, and believe that simple firewall and other access controls can mitigate most of those vectors. This is very much a defense-in-depth strategy, so I am only presenting a small piece of that puzzle here.
tw04 5y ago
You're kind of missing the point. The whole point of data at rest encryption is: if this drive dies and gets thrown into a dumpster, I don't want some random shmuck to pull it out, put it in a server, run a recovery program against it, and get my customer's SSNs. Now, I would hope Amazon shreds drives that are declared dead, but I wouldn't want to risk my business on it. It has almost nothing to do with protection against a targeted attack and everything to do with chain of custody.

CamouflagedKiwi 5y ago

Was hoping for something a little more profound than "use an encrypted storageclass for your volumes".

Fiahil 5y ago
Yes, what they explained is entirely based on encrypted EBS volume. Nothing related with Kubernetes apart from the ˋencrypted: true` in the StorageClass.

sylr 5y ago

Am I the only one enforcing a strict no database in kubernetes policy ?

plaur782 5y ago
No. You are not alone. There are still a number of organizations that are cautious about deploying databases in Kubernetes. That said, various third party surveys as well as anecdotal evidence of what we are seeing at Crunchy Data suggests that deploying databases on Kubernetes is increasingly common. The degree to which it makes sense often depends on whether or not the organization is standardizing around Kubernetes for their deployment model more generally.
zinclozenge 5y ago
You're not the only one, obviously. But that stance is not really relevant anymore. Just about every database as a service company will run their databases in kubernetes now.
Sebb767 5y ago
Honest question: Why is that? What is wrong with your DB running in Kubernetes and what do you suggest as alternative?
- sylr 5y ago
  I consider anything in kubernetes disposable. I want to be able to lift and shift anything in a new cluster without any migration in a matter of minutes. Anything "stateful" like a database breaks this paradigm. I have nothing against databases used as cache that can be "re-filled" upon re-creation, but I believe anything holding business critical data shall be held outside of a kubernetes cluster. Why, because being one command away deleting your StatefulSet, Helm Release ... etc scares the shit out of me. You can of course minimize the risk with correct RBAC, ensure proper backup/restore migrations but that require lots of staff and efforts I can't spare. So until I can be reassured that I have all the tooling that can recover rapidly any catastrophic failure/mishap, and that all this tooling is tested monthly, I enforce using managed databases services.

handrous 5y ago

Tangentially related: what's state-of-the-art for data protection & access control for small organizations? One runs into the "someone's gotta be trusted with the master keys" problem there so early & often that all the "big" solutions feel silly. Do small shops just farm this out via SaaS and hope their provider's doing the right thing? (the answer back in the day, and perhaps still, was just "they don't really worry about it at all, and hope nothing goes wrong")

ianlevesque 5y ago

This post was really laying it on a bit thick on the marketing, with three mentions of their own products before even finishing the introduction. I know that’s the point of most of these posts but then when the content was also a product, it’s too much. Pass.