6 ms·

Shades of the AOL search data leak: https://en.wikipedia.org/wiki/AOL_search_data_leak https://en.wikipedia.org/wiki/AOL_search_data_leak Of course personally

Shades of the AOL search data leak: https://en.wikipedia.org/wiki/AOL_search_data_leak https://en.wikipedia.org/wiki/AOL_search_data_leak Of course personally identifiable information will be extracted despite this model. "Differential Privacy" is cynical academic malpractice -- selling a reputation so that when individuals are harmed in the course of commercial exploitation of the purportedly anonymized data, the organizations that profited can avoid being held responsible. We never learn, because there is money to be made if we pretend that anonymization works.

Hoobentube 9y ago
I confused, you sound against the idea of differential privacy, even though the foundation of differential privacy is that anonymization DOESN'T work to protect people's privacy. In fact, the canonical example used in the differential privacy field of failed anonymization is the AOL fiasco.
Jtsummers 9y ago
That data (AOL search data) wasn't intentionally released so while pseudonyms had been used, no serious effort at fully (or more fully) anonymizing or protecting users had occurred. Here the intent is to be able to produce data sets where individual identification will be statistically unlikely if not impossible (by fuzzing the data), or where individuals can refute the data because there's a statistical chance the data is a lie (probability biased in favor of truth so that the aggregate data is still useful).
AdamSC1 9y ago
To be clear, I think you're right that no tracking is better than trying to protect data. However, it's important to understand that 'anonymization' is very different than the practice of "Differential Privacy." I'm no expert but here is how I understand it as a simplified example: Imagine your information is stored in a spreadsheet. It is storing your weight, height, zipcode, age and name. The 'anonymization' spreadsheet would still have a unique row dedicated to you (similar to a spreadsheet) and it may replace your name with an ID# or an encrypted string. Now, just like in the AOL dataleak that information being stored as a single line item is still easy to backtrack as there is likely no one else with your weight, height and age combination in your zipcode. So a hacker can identify a single person. Differential Privacy would store information differently, perhaps in separate spreadsheets, one that is list of heights, one that is a list of weights, etc, etc. No two spreadsheets would store the information in the same order (#3 on the height list would not be #3 on the weight list) and it may even contain some incorrect dummy information. There would be some sort of algorithmic relation however that allows a system to create outputs in which the data has meaningful information (trends, means, standard deviations etc) but it can not be back-tracked to identify any single unique row. Differential Privacy allows us to see the trend "Males age 45 are taller on average than Females age 45" but not say "User #155083 is age 45, weighs 195lbs, and lives in zipcode 10001" That's a big difference in privacy, and while it isn't perfect it is a step in the right direction. While I wish more companies would adopt a no-data policy, it is at least better that they are responsible as can be with the data they have.
- rectang 9y ago
  Such obscuring is vulnerable against sidechannel attacks to re-link the record fragments. For example, misspellings, incidental geographic information, topics -- any pattern which is unusual, not anticipated by the modeler and deliberately obliterated. What links the AOL fiasco and this one is that both believe they have thought of everything important. They're wrong -- and there will always be a future attacker to prove it. You can't fight information theory. Differential Privacy is an excuse to get around sensible no-data policies -- by making irresponsible promises, it will result in more privacy violations, not less.
  - AdamSC1 9y ago
    You're right these systems are currently flawed, and may remained flawed. But, the same is true for all systems. Don't get me wrong I agree in theory that every company should not retain any customer information - I work for one of a few companies that does that, but, I also know that many company's will not make that switch (example a bank, or a medical record) and in such cases that data should be as secure as possible. To counter the irresponsible promises you are talking about it'd be ideal to see compliance and security regulation like we see with HIPPA applied to all customer data, but until we evolve proper trust-less identification models and ways for users to self-secure and trustlessly validate their information then some businesses will always collect data.
  - frankmcsherry 9y ago
    > You can't fight information theory. I totally agree with this statement, but I think you are confused about differential privacy. Its guarantees are information theoretic (specifically, a bound on the relative Bayes factors of any conclusion, with and without any individual record). You are obviously welcome to be skeptical, but much of what you've posted so far is not correct.
    - rectang 9y ago
      Of course I don't dispute the math. I maintain that these guarantees will not be achieved in practice because they rely on impossibly airtight implementation and impossibly omniscient modeling.
specialist 9y ago
I share your cynicism. I did a lot of work in healthcare IT and have also obsessed about voter privacy (protecting the secret ballot). I've seen de-anonymization in practice. I also know there's a huge chasm between best available science (and practices) vs the real world. (Dumb example: mis-redacting PDFs with opaque boxes instead of removing the text.) #1 At best, like crypto, differential privacy may offer temporary protection, to data re-used (shared, in transit), if given assumptions are preserved. #2 Also, like crypto, I have no confidence that anyone, any where will implement DP correctly, or even be able to prove they've done it correctly. #3 The original data is stored somewhere. There is no DP story for mitigating leaks. Given my disappointment, I believe (but cannot prove) there's two worthwhile strategies worth exploring. First, contracts to use case box and time box the data, stating how and when shared data may be used, and then a drop dead time when shared data must be destroyed. Part of this contract could be expanded to include parameters for differential privacy. One org I work with has these policies. Alas, Thatcherite "trust, but verify" is tough. We add fake data (honeypot-esque) and have caught cheaters. Second, I'm keen to further explore translucent databases, where data in situ (at rest) is encrypted. Lastly, I'm always looking to see who is working in this space, and what they're doing. I'd like to believe that someone will crack this nut.

Hoobentube 9y ago

I confused, you sound against the idea of differential privacy, even though the foundation of differential privacy is that anonymization DOESN'T work to protect people's privacy. In fact, the canonical example used in the differential privacy field of failed anonymization is the AOL fiasco.

Jtsummers 9y ago

That data (AOL search data) wasn't intentionally released so while pseudonyms had been used, no serious effort at fully (or more fully) anonymizing or protecting users had occurred. Here the intent is to be able to produce data sets where individual identification will be statistically unlikely if not impossible (by fuzzing the data), or where individuals can refute the data because there's a statistical chance the data is a lie (probability biased in favor of truth so that the aggregate data is still useful).

AdamSC1 9y ago

To be clear, I think you're right that no tracking is better than trying to protect data. However, it's important to understand that 'anonymization' is very different than the practice of "Differential Privacy." I'm no expert but here is how I understand it as a simplified example: Imagine your information is stored in a spreadsheet. It is storing your weight, height, zipcode, age and name. The 'anonymization' spreadsheet would still have a unique row dedicated to you (similar to a spreadsheet) and it may replace your name with an ID# or an encrypted string. Now, just like in the AOL dataleak that information being stored as a single line item is still easy to backtrack as there is likely no one else with your weight, height and age combination in your zipcode. So a hacker can identify a single person. Differential Privacy would store information differently, perhaps in separate spreadsheets, one that is list of heights, one that is a list of weights, etc, etc. No two spreadsheets would store the information in the same order (#3 on the height list would not be #3 on the weight list) and it may even contain some incorrect dummy information. There would be some sort of algorithmic relation however that allows a system to create outputs in which the data has meaningful information (trends, means, standard deviations etc) but it can not be back-tracked to identify any single unique row. Differential Privacy allows us to see the trend "Males age 45 are taller on average than Females age 45" but not say "User #155083 is age 45, weighs 195lbs, and lives in zipcode 10001" That's a big difference in privacy, and while it isn't perfect it is a step in the right direction. While I wish more companies would adopt a no-data policy, it is at least better that they are responsible as can be with the data they have.

rectang 9y ago
Such obscuring is vulnerable against sidechannel attacks to re-link the record fragments. For example, misspellings, incidental geographic information, topics -- any pattern which is unusual, not anticipated by the modeler and deliberately obliterated. What links the AOL fiasco and this one is that both believe they have thought of everything important. They're wrong -- and there will always be a future attacker to prove it. You can't fight information theory. Differential Privacy is an excuse to get around sensible no-data policies -- by making irresponsible promises, it will result in more privacy violations, not less.
- AdamSC1 9y ago
  You're right these systems are currently flawed, and may remained flawed. But, the same is true for all systems. Don't get me wrong I agree in theory that every company should not retain any customer information - I work for one of a few companies that does that, but, I also know that many company's will not make that switch (example a bank, or a medical record) and in such cases that data should be as secure as possible. To counter the irresponsible promises you are talking about it'd be ideal to see compliance and security regulation like we see with HIPPA applied to all customer data, but until we evolve proper trust-less identification models and ways for users to self-secure and trustlessly validate their information then some businesses will always collect data.
- frankmcsherry 9y ago
  > You can't fight information theory. I totally agree with this statement, but I think you are confused about differential privacy. Its guarantees are information theoretic (specifically, a bound on the relative Bayes factors of any conclusion, with and without any individual record). You are obviously welcome to be skeptical, but much of what you've posted so far is not correct.
  - rectang 9y ago
    Of course I don't dispute the math. I maintain that these guarantees will not be achieved in practice because they rely on impossibly airtight implementation and impossibly omniscient modeling.

specialist 9y ago

I share your cynicism. I did a lot of work in healthcare IT and have also obsessed about voter privacy (protecting the secret ballot). I've seen de-anonymization in practice. I also know there's a huge chasm between best available science (and practices) vs the real world. (Dumb example: mis-redacting PDFs with opaque boxes instead of removing the text.) #1 At best, like crypto, differential privacy may offer temporary protection, to data re-used (shared, in transit), if given assumptions are preserved. #2 Also, like crypto, I have no confidence that anyone, any where will implement DP correctly, or even be able to prove they've done it correctly. #3 The original data is stored somewhere. There is no DP story for mitigating leaks. Given my disappointment, I believe (but cannot prove) there's two worthwhile strategies worth exploring. First, contracts to use case box and time box the data, stating how and when shared data may be used, and then a drop dead time when shared data must be destroyed. Part of this contract could be expanded to include parameters for differential privacy. One org I work with has these policies. Alas, Thatcherite "trust, but verify" is tough. We add fake data (honeypot-esque) and have caught cheaters. Second, I'm keen to further explore translucent databases, where data in situ (at rest) is encrypted. Lastly, I'm always looking to see who is working in this space, and what they're doing. I'd like to believe that someone will crack this nut.