4 ms·

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor perf

Per user encryption such as you suggest does not address the problem. Systems that work this way have existed for decades and they have pathologically poor performance and scalability. Many classes of major optimization in databases don't work under these constraints. This isn't a novel idea, having been implemented for decades; it has been broadly rejected because it doesn't actually work in real systems without tradeoffs no one accepts, even in highly security sensitive environments. You are overly dismissive of the technical challenges of actually deleting data, particularly in large systems. Technically naive solutions like having separate encryption keys for individual entities sounds good but it doesn't actually address the problem.

coder543 4y ago
I feel like you’re overly dismissive of the other things I mentioned. Database failover comes with inconveniences, but it does allow you to use a single key and get great performance. For massive databases, it might be impossible to do that way, but most companies don’t have massive databases. I also opened a question about how ”erased” deleted data needs to be to be compliant, since I would imagine a lot of companies consider deletion on an encrypted drive to be good enough, even if the data may theoretically be recoverable under extreme circumstances until that hard drive is completely overwritten multiple times or destroyed. Per user keys are also very useful for use cases like a personal file storage service, even if traditional RDBMS don’t work well with them. Techniques are situationally dependent. I was not being dismissive of the existence of technical challenges, but rather dismissive of comments like: > To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes without choosing significantly unorthodox hardware (or destroying a drive every time you need a single bit erased), no matter how much you want to in software. Which seem to completely deride the possibility of compliance. Compliance is feasible, even if it has challenges.
- heisjustsosmart 4y ago
  Save yourself a lot of time, read jandrewrogers comments before responding to him.
  - deleted 4y ago
    [deleted]
  - dcow 4y ago
    coder543 initially responded to dataflow. Then jandrewrogers included an additional anecdote, but only in a single comment. As the thread stands at the time I posed this, jandrewrogers does not have multiple comments.
    - heisjustsosmart 4y ago
      An ambiguous phrase - I was not criticizing you but only advising you read his historical statements on databases before engaging. He has wasted a great deal of time of other people.
      - dcow 4y ago
        Do you know how to read usernames?
        heisjustsosmart 4y ago
        Yes. Your point being what.
        dcow 4y ago
        I am not the person you originally replied to but you (again) commented like I was. My comment was about how your comment erroneously targets the wrong person because you thought you were replying to one person when you were replying to another. You did it again in your previous comment hence my serious question.
        heisjustsosmart 4y ago
        It doesn't matter, it was advice for all.
dcow 4y ago
Is there literature on this topic? I'd like to learn at what scale per-user/tenant keys becomes untenable and the characteristics of systems that exhibit pathologically poor performance due to such a design. Naively, it sounds like individual entity keys does solve the problem of deletion, but that your argument is that the tradeoffs aren't generally worth it?
- nicoburns 4y ago
  I’d imagine a pretty small scale. Imagine an endpoint that lists user accounts. You would have to individually decrypt each row! and you would never be able to do database joins of data in these tables because you it wouldn’t be possible to decrypt it until it had already left the database.
closewith 4y ago
With respect, your own comment reads as overly dismissive of the regulatory environment introduced by the GDPR. Systems (and companies) have for decades collected and stored personal data with little or no regard for the lifecycle of the data. Since the introduction of the Regulation, companies must now refrain from the collection of personal data if they cannot satisfy the rights of the data subject, including the right to erasure (which is obviously not an absolute right). If the technical solutions aren't performant or scalable enough for data controllers, then the result is that they cannot collect personal data. The idea that companies have in the past rejected solutions which protect the rights of data subjects because they were unwilling to accept the technical tradeoffs is exactly why strong regulation such as the GDPR is required. Frankly, your comment reads as contemptuous to the rights of the end-user. Thankfully, the EU offers protection to its residents from people with that attitude.

coder543 4y ago

I feel like you’re overly dismissive of the other things I mentioned. Database failover comes with inconveniences, but it does allow you to use a single key and get great performance. For massive databases, it might be impossible to do that way, but most companies don’t have massive databases. I also opened a question about how ”erased” deleted data needs to be to be compliant, since I would imagine a lot of companies consider deletion on an encrypted drive to be good enough, even if the data may theoretically be recoverable under extreme circumstances until that hard drive is completely overwritten multiple times or destroyed. Per user keys are also very useful for use cases like a personal file storage service, even if traditional RDBMS don’t work well with them. Techniques are situationally dependent. I was not being dismissive of the existence of technical challenges, but rather dismissive of comments like: > To be honest, in the age of modern overprovisioned storage drives that remap blocks frequently, I'm not really sure you can implement genuine "hard" deletes without choosing significantly unorthodox hardware (or destroying a drive every time you need a single bit erased), no matter how much you want to in software. Which seem to completely deride the possibility of compliance. Compliance is feasible, even if it has challenges.

heisjustsosmart 4y ago
Save yourself a lot of time, read jandrewrogers comments before responding to him.
- deleted 4y ago
  [deleted]
- dcow 4y ago
  coder543 initially responded to dataflow. Then jandrewrogers included an additional anecdote, but only in a single comment. As the thread stands at the time I posed this, jandrewrogers does not have multiple comments.
  - heisjustsosmart 4y ago
    An ambiguous phrase - I was not criticizing you but only advising you read his historical statements on databases before engaging. He has wasted a great deal of time of other people.
    - dcow 4y ago
      Do you know how to read usernames?
      - heisjustsosmart 4y ago
        Yes. Your point being what.
        dcow 4y ago
        I am not the person you originally replied to but you (again) commented like I was. My comment was about how your comment erroneously targets the wrong person because you thought you were replying to one person when you were replying to another. You did it again in your previous comment hence my serious question.
        heisjustsosmart 4y ago
        It doesn't matter, it was advice for all.

dcow 4y ago

Is there literature on this topic? I'd like to learn at what scale per-user/tenant keys becomes untenable and the characteristics of systems that exhibit pathologically poor performance due to such a design. Naively, it sounds like individual entity keys does solve the problem of deletion, but that your argument is that the tradeoffs aren't generally worth it?

nicoburns 4y ago
I’d imagine a pretty small scale. Imagine an endpoint that lists user accounts. You would have to individually decrypt each row! and you would never be able to do database joins of data in these tables because you it wouldn’t be possible to decrypt it until it had already left the database.

closewith 4y ago

With respect, your own comment reads as overly dismissive of the regulatory environment introduced by the GDPR. Systems (and companies) have for decades collected and stored personal data with little or no regard for the lifecycle of the data. Since the introduction of the Regulation, companies must now refrain from the collection of personal data if they cannot satisfy the rights of the data subject, including the right to erasure (which is obviously not an absolute right). If the technical solutions aren't performant or scalable enough for data controllers, then the result is that they cannot collect personal data. The idea that companies have in the past rejected solutions which protect the rights of data subjects because they were unwilling to accept the technical tradeoffs is exactly why strong regulation such as the GDPR is required. Frankly, your comment reads as contemptuous to the rights of the end-user. Thankfully, the EU offers protection to its residents from people with that attitude.