6 ms·

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hop

Author Here. Using crypto hashes to anonymize data is one of those mistakes I've seen several times, and wanted to draw some attention to the issue so that hopefully we can all learn from it. Let me know if you have any questions.

deleted 8y ago
[deleted]
oleganza 8y ago
Have you looked into VRFs (verifiable random functions)? This is essentially a public key crypto version of HMACs. You have a piece of data and a blinding factor, both are committed into a single point on an elliptic curve, and blinding factor is not exposed as-is (as in case with salt for hashed content), but instead committed into a public key (a point). The resulting point preserves the algebraic structure, which allows you to sign proofs about it, but w/o leaking the blinding factor. You can even sign a proof to a "designated verifier", which makes the proof useful only to a specific party, but anyone else wouldn't be able to trust it as it could be forged by that other party.
- kevin_nisbet 8y ago
  No I haven't, this sounds very intriguing though, I'll have to do some research.
sdenton4 8y ago
So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-way hashing to the random internal UUID, as well: for example, to provide different levels of logs access to different internal users. People who make dashboards get hashed ids, and people who debug logging get raw ids.) The problem of entropy allowing individual user identification even with all IDs scrubbed is still very real, though, and non-trivial to undertake. One can start by wrapping the query engine with a service which checks that a certain minimum number of people are covered by a given query before returning the results. Or apply differential privacy-type transformations to the output...
- kevin_nisbet 8y ago
  This is along the lines of where I was going with the alternative approach, I just simplified it for brevity. :) In the case of teleport, I think this is a bit more difficult to achieve, because we don't necessarily have our own account database, our common commercial use case is integrated to an identity provider through SAML/OIDC, which I'm not sure would consistently offer a random id per account to use. While there are many way's we could generate and store the username <-> random id mappings, this adds a certain amount of complexity to get right on a distributed system. If building a system from scratch with end to end control, I do prefer the random identifier approach.
- merinowool 8y ago
  Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.
  - unilynx 8y ago
    If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.
    - JumpCrisscross 8y ago
      > If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.
      - GordonS 8y ago
        For all practical purposes, a secure, one-way cryptographic hash is irreversible.
        shabble 8y ago
        I'm thinking of a number between 1 and 100. It's bcrypt hash is: '$2b$15$qUxzZ5ZF55lMuqiH9GMjQOHkNyee86qd2Vh2kQyF5P3U6JZJx9AEC' I bet nobody could ever reverse this secure cryptographic hash to figure out what it could be... ;)
EGreg 8y ago
I think this is the best advice: “And most importantly, we only collect enough data to fulfill our stated purpose. The fewer data points that we collect, the less opportunity that someone can correlate the data.” The smaller the domain, the less anonymization works to conceal which user id did an action. However, if you think about it, identity is far more than user id. That is why real anonymity means not storing any unnecessary information from other domains. We have a technique where we use iframes to display a person’s name, friends etc. back to them based on user ids, but the enclosing domain knows only the user ids and their connections.
mjevans 8y ago
Technically a substitution lookup table (like is proposed at the end) is analogous to one-time-pad 'encryption'. In this case the 'pad' is only used within a single (extended time domain) context and is presumably selectively exposed and used only in a contexts where intercepting the context already reveals that data anyway. Additional security could be added by making a session-unique identifier (not based on user, chronological, or external context data) and only having the master lookup table for user to sessions in an elevated security environment.
- cvwright 8y ago
  Yeah, it sounds a lot like deterministic encryption. Which is OK if your input data is essentially uniform random. But if it's not, and your adversary knows the distribution of the input data, then the protection level is pretty close to zero.
polskibus 8y ago
Please elaborate on whether hashing is a good pseudonimization strategy in context of GDPR guidelines.
- js8 8y ago
  I thought the blog post made it pretty clear that it isn't.
  - jack9 8y ago
    The blog post wasnt about > pseudonimization strategy in context of GDPR guidelines. i.e. Doing the minimal work to meet the guidelines. Effectiveness is incidental to that goal. Maybe I misunderstood the GP tho.

deleted 8y ago

[deleted]

oleganza 8y ago

Have you looked into VRFs (verifiable random functions)? This is essentially a public key crypto version of HMACs. You have a piece of data and a blinding factor, both are committed into a single point on an elliptic curve, and blinding factor is not exposed as-is (as in case with salt for hashed content), but instead committed into a public key (a point). The resulting point preserves the algebraic structure, which allows you to sign proofs about it, but w/o leaking the blinding factor. You can even sign a proof to a "designated verifier", which makes the proof useful only to a specific party, but anyone else wouldn't be able to trust it as it could be forged by that other party.

kevin_nisbet 8y ago
No I haven't, this sounds very intriguing though, I'll have to do some research.

sdenton4 8y ago

So one of the issues here is using an externally visible ID (or a transformation of such) as an internal ID. Why not create a random int64 at account creation time which is invisibly linked to the public username (eg, email address). So now you've got a proper join key, you can restrict access to the map, and it's easy to delete the map entry when the user unsubscribes. (There can still be good reasons to apply one-way hashing to the random internal UUID, as well: for example, to provide different levels of logs access to different internal users. People who make dashboards get hashed ids, and people who debug logging get raw ids.) The problem of entropy allowing individual user identification even with all IDs scrubbed is still very real, though, and non-trivial to undertake. One can start by wrapping the query engine with a service which checks that a certain minimum number of people are covered by a given query before returning the results. Or apply differential privacy-type transformations to the output...

kevin_nisbet 8y ago
This is along the lines of where I was going with the alternative approach, I just simplified it for brevity. :) In the case of teleport, I think this is a bit more difficult to achieve, because we don't necessarily have our own account database, our common commercial use case is integrated to an identity provider through SAML/OIDC, which I'm not sure would consistently offer a random id per account to use. While there are many way's we could generate and store the username <-> random id mappings, this adds a certain amount of complexity to get right on a distributed system. If building a system from scratch with end to end control, I do prefer the random identifier approach.
merinowool 8y ago
Then user emails you to ask what personal data of his you have on the server. Now you don't have a connection so you can't find it, but you have it. GDPR non compliance.
- unilynx 8y ago
  If you can't connect it to the user in any way, it's no longer personal information. Expect the data protection agency to compliment you.
  - JumpCrisscross 8y ago
    > If you can't connect it to the user in any way, it's no longer personal information Just because you can't connect it doesn't mean nobody else can.
    - GordonS 8y ago
      For all practical purposes, a secure, one-way cryptographic hash is irreversible.
      - shabble 8y ago
        I'm thinking of a number between 1 and 100. It's bcrypt hash is: '$2b$15$qUxzZ5ZF55lMuqiH9GMjQOHkNyee86qd2Vh2kQyF5P3U6JZJx9AEC' I bet nobody could ever reverse this secure cryptographic hash to figure out what it could be... ;)

EGreg 8y ago

I think this is the best advice: “And most importantly, we only collect enough data to fulfill our stated purpose. The fewer data points that we collect, the less opportunity that someone can correlate the data.” The smaller the domain, the less anonymization works to conceal which user id did an action. However, if you think about it, identity is far more than user id. That is why real anonymity means not storing any unnecessary information from other domains. We have a technique where we use iframes to display a person’s name, friends etc. back to them based on user ids, but the enclosing domain knows only the user ids and their connections.

mjevans 8y ago

Technically a substitution lookup table (like is proposed at the end) is analogous to one-time-pad 'encryption'. In this case the 'pad' is only used within a single (extended time domain) context and is presumably selectively exposed and used only in a contexts where intercepting the context already reveals that data anyway. Additional security could be added by making a session-unique identifier (not based on user, chronological, or external context data) and only having the master lookup table for user to sessions in an elevated security environment.

cvwright 8y ago
Yeah, it sounds a lot like deterministic encryption. Which is OK if your input data is essentially uniform random. But if it's not, and your adversary knows the distribution of the input data, then the protection level is pretty close to zero.

polskibus 8y ago

Please elaborate on whether hashing is a good pseudonimization strategy in context of GDPR guidelines.

js8 8y ago
I thought the blog post made it pretty clear that it isn't.
- jack9 8y ago
  The blog post wasnt about > pseudonimization strategy in context of GDPR guidelines. i.e. Doing the minimal work to meet the guidelines. Effectiveness is incidental to that goal. Maybe I misunderstood the GP tho.