8 ms·

The whole anonymization of IP addresses by just hashing the date and IP is just security theater. Cryptographic hashes are designed to be fast. You can do 6 bi

• berkes 3y ago
  Maybe they use a secret salt or rotating salt? The example code doesn't, so I'm afraid you are right. But one addition and it can be made reasonable secure. I am afraid, however, that this security theater is enough to pass many laws, regulations and such on PII.
• ktta 3y ago
  Not if they use a password hash like Argon2 or scrypt
  • ale42 3y ago
    But that's very heavy to compute at scale...
    • isodev 3y ago
      True, but also it's a blogging platform - does it really have that kind of scale to be concerned with?
      • ale42 3y ago
        Probably not, I was mainly thinking if that kind of solution was to be adopted at a scale like Google Analytics.
  • __alexs 3y ago
    Even then it is theatre because if you know the IP address you want to check it's trivial to see if there's a match.
    • chrismorgan 3y ago
      And this is why such a hash will still be considered personal data under legislation like GDPR.
• TekMol 3y ago
  That is easy to fix though. Just use a temporary salt. Pseudo code: if salt.day < today(): salt = {day: today(), salt: random()} ip_hash = sha256(ip + salt.salt)
  • __alexs 3y ago
    Assuming you don't store the salts, this produces a value that is useless for anything but counting something like DAU. Which you could equally just do by counting them all and deleting all the data at the end of the day, or using a cardinality estimator like HLL.
    • TekMol 3y ago
      DAU in regards to a given page. Have you read the article? That is what the author's goal seems to be. He wants to prevent multiple requests to the same page by the same IP counted multiple times.
      • tatersolid 3y ago
        Is that more efficiently done with an appropriate caching header on the page as it is served? Cache-Control: private, max-age=86400 This prevents repeat requests for normal browsers from hitting the server.
    • dvdkon 3y ago
      That same uselessness for long-term identification of users is what makes this approach compliant with laws regulating use of PII, since what you have after a small time window isn't actually PII (unless correlated with another dataset, but that's always the case).
    • SamBam 3y ago
      That's precisely all that OP is storing in the original article. They're just getting a list of hashes per day, and associated client info. They have no idea if the same user visit them on multiple days, because the hashes will be different.
  • 3y ago
• Etheryte 3y ago
  For context, this problem also came up in a discussion about Storybook doing something similar in their telemetry [0] and with zero optimization it takes around two hours to calculate the salted hashes for every IPv4 on my home laptop. [0] https://news.ycombinator.com/item?id=37596757 https://news.ycombinator.com/item?id=37596757
  • deleted 3y ago
    [deleted]
• hnreport 3y ago
  This the type of comment that reinforces not even trying to learn or outsource security. You’ll never know enough.
  • petesergeant 3y ago
    I think the opposite? I’m a dev with a bit of an interest in security, and this immediately jumped out at me from the story; knowing enough security to discard bad ideas is useful.
• WhyNotHugo 3y ago
  Aside from it being technically trivial to get an IP back from its hash, the EU data protection agency made it very clear that "hashing PII does not count as anonymising PII". Even if you hash somebody's full name, you can later answer the question "does this hash match the this specific full name". Being able to answer this question implies that the anonymisation process is reversible.
  • kevincox 3y ago
    I think the word "reversible" here is being stretched a bit. There is a significant difference between being able to list every name that has used your service and being able to check if a particular name has used your service. (Of course these can be effectively the same in cases where you can list all possible inputs such as hashed IPv4 addresses.) That doesn't mean that hashing is enough for pure anonymity, but used properly hashes are definitely a step above something fully reversible (like encryption with a common key).
    • deleted 3y ago
      [deleted]
    • SamBam 3y ago
      I'm not sure the distinction is meaningful. If the police demand your logs to find out whether a certain IP address visited in the past year, they'd be able to find that out pretty quickly given what's stored. So how is privacy being respected?
    • pluto_modadic 3y ago
      if it fulfills the same function, does it matter? if you have an ad ID for a person, say example@example.com, and you want to deduplicate it, if you provide them with the names, the company that buys the data can still "blend" it with data they know, if they know how the hash was generated... and effectively get back that person's email, or IP, or phone number, or at least get a good hunch that the closest match is such and such person with uncanny certainty de-anonymization of big data is trivial in basically every case that was written by an advertising company, instead of written by a truly privacy focused business. if it were really a non-reversible hash, it would be evenly distributed, not predictable, and basically useless for advertising, because it wouldn't preserve locality. It needs to allow for finding duplicates... so the person you give the hash to, can abuse that fact.
• alkonaut 3y ago
  Hashes should be salted. If you salt, you are fine, if you don't you aren't. Whether the salt can be kept indefinitely, or is rotated regularly etc is just an implementation detail, but the key with salting hashes for analytics is that the salt never leaves the client. As explained in the article there seems to be no salt (or rather, the current date seems to be used as a salt, but that's not a random salt and can easily be guessed for anyone who wants to say "did IP x.y.z.w visit on date yy-mm-dd?". It's pretty easy to reason about these things if you look from the perspective of an attacker. How would you do to figure out anything about a specific person given the data? If you can't, then the data is probably OK to store.
  • piaste 3y ago
    > Hashes should be salted. If you salt, you are fine, if you don't you aren't. > Whether the salt can be kept indefinitely, or is rotated regularly etc is just an implementation detail, but the key with salting hashes for analytics is that the salt never leaves the client. I think I'm missing something. If the salt is known to the server, then it's useless for this scenario. Because given a known salt, you can generate the hashes for every IP address + that salt very quickly. (Salting passwords works because the space for passwords is big, so rainbow tables are expensive to generate.) If the salt is unknown to the server, i.e. generated by the client and 'never leaves the client'... then why bother with hashes? Just have the client generate a UUID directly instead of a salt.
    • rkangel 3y ago
      Without a salt, you can generate the hash for every IP address once, and then permanantly have a hash->IP lookup (effectively a Rainbow table). If you have a salt, then you need to do it for each database entry, which does make it computationally more expensive.
      • tptacek 3y ago
        People are obsessed with this attack from the 1970s, but in practice password cracking rigs just brute force the hashes, and that has been the practice since my career started in the 1990s and people used `crack`, into the 2000s and `jtr`, and today with `hashcat` or whatever it is the cool kids use now. "Rainbow tables" don't matter. If you're discussing the expense of attacking your scheme with or without rainbow tables, you've already lost.
• deleted 3y ago
  [deleted]
• dspillett 3y ago
  > Cryptographic hashes are designed to be fast. Not really. They are designed to be fast enough and even then only as a secondary priority. > You can do 6 billion … hashes/second on [commodity hardware] … there’s only 4 billion ipv4 addresses. So you can brute force the entire range This is harder if you use a salt not known to the attacker. Per-entry salts can help even more, though that isn't relevant to IPv4 addresses in a web/app analytics context because after the attempt at anonymisation you want to still be able to tell that two addresses were the same. > And that’s true even if they used something secure like SHA-256 instead of broken MD5 Relying purely on the computation complexity of one hash operation, even one not yet broken, is not safe given how easy temporary access to mass CPU/GPU power is these days. This can be mitigated somewhat by running many rounds of the hash with a non-global salt – which is what good key derivation processes do for instance. Of course you need to increase the number of rounds over time to keep up with the rate of growth in processing availability, to keep undoing your hash more hassle than it is worth. But yeah, a single unsalted hash (or a hash with a salt the attacker knows) on IP address is not going to stop anyone who wants to work out what that address is.
  • krsdcbl 3y ago
    Don't forget that md5 is comparatively slow & there are way options for hashing nowadays: https://jolynch.github.io/posts/use_fast_data_algorithms/ https://jolynch.github.io/posts/use_fast_data_algorithms/
  • SAI_Peregrinus 3y ago
    A "salt not known to the attacker" is a "key" to a keyed hash function or message authentication code. A salt isn't a secret, though it's not usually published openly.
  • marcosdumay 3y ago
    > only as a secondary priority That's not a reasonable way to say it. It's literally the second priority, and heavily evaluated when deciding what algorithms to take. > This is harder if you use a salt not known to the attacker. The "attacker" here is the sever owner. So if you use a random salt and throw it away, you are good, anything resembling the way people use salt on practice is not fine.
• HermanMartinus 3y ago
  Author here. I commented down below, but it's probably more relevant in this thread. For a bit of clarity around IP addresses hashes. The only use they have in this context is preventing duplicate hits in a day (making each page view unique by default). At the end of each day there is a worker job that scrubs the ip hash which is now irrelevant.
  • myfonj 3y ago
    Have you considered serving actual small transparent image with caching headers set to expire at midnight?