5 ms·

Great point! We wrote about how we're doing it on whotracks.me a while ago, if anyone is wondering how that could work: https://whotracks.me/blog/private_analyt

by pythux 7y ago

Great point! We wrote about how we're doing it on whotracks.me a while ago, if anyone is wondering how that could work: https://whotracks.me/blog/private_analytics.html https://whotracks.me/blog/private_analytics.html

marcus_holmes 7y ago
this is useful, thanks... couple questions: 1. Do you grab the country from the IP address before anonymising it? (I know VPN's are a thing, but we would like to know where our customers are coming from) 2. Why encrypt with a daily key instead of just hashing the IP address and storing the hash? Do you ever decrypt the IP address? 3. How have you found CloudWatch? I'm logging all requests to our own database, because it's so simple to do. What's the bonus for Cloudwatch?
- kkm 7y ago
  Hi Marcus, Thank you for your questions. Disclaimer: I also work on WTM from time to time, so I hope can answer these questions to some degree :) 1. WhoTracksMe is served via CloudFront, it only get's the IP that CF reports. To get the country, there are two ways that can be used. IP => Country mapping via a Database or additionally, because CF has multiple edge locations, if needed the you could see which edge location served the request, that will help you narrow down the location at a country level. Ofcourse, in both cases if the user is using VPN, you would only records the VPN location. 2. The problem we want to avoid is same IP with same hashing algorithm will produce same value. Which can then be used to co-relate user activity across days. Second, depending on what hashing scheme one is using, rainbow tables can be used to get back the original value. Therefore, to avoid, we used the approach of daily key. Now that I write, we could also use some hash + daily_salt. This should give the same guarantees. - Needs to be checked. 3. In this setup we are not using Cloudwatch but Cloudfront, which is the CDN provider from AWS.
  - marcus_holmes 7y ago
    1. Ah, OK, that's useful. I was thinking that dropping the last byte of the IP address would still be able to give us the country via database lookup, but anonymise enough. But getting it via CF (or similar) would be a good alternative. 2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant). But adding a daily salt would decrease the odds, for sure. 3. Sorry, my bad, I meant Cloudfront, but it's Sunday night here and there has been wine ;) Same question: how has Cloud{{thing}} worked out for you? Is it worth the extra complexity?
    - FabHK 7y ago
      > 2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant) The problem, as I understand GP, is not so much the fear that two different IPs might collide. The problem is that seeing the same hash, you know that it refers to the same IP, and as such you can correlate users/sites over time (a privacy invasion).
      - marcus_holmes 7y ago
        That makes sense, kinda. Though I'm a little unsure why correlating http requests over an hour is not invading privacy, but doing the same over a week is?

by pythux 7y ago

marcus_holmes 7y ago

this is useful, thanks... couple questions: 1. Do you grab the country from the IP address before anonymising it? (I know VPN's are a thing, but we would like to know where our customers are coming from) 2. Why encrypt with a daily key instead of just hashing the IP address and storing the hash? Do you ever decrypt the IP address? 3. How have you found CloudWatch? I'm logging all requests to our own database, because it's so simple to do. What's the bonus for Cloudwatch?

kkm 7y ago
Hi Marcus, Thank you for your questions. Disclaimer: I also work on WTM from time to time, so I hope can answer these questions to some degree :) 1. WhoTracksMe is served via CloudFront, it only get's the IP that CF reports. To get the country, there are two ways that can be used. IP => Country mapping via a Database or additionally, because CF has multiple edge locations, if needed the you could see which edge location served the request, that will help you narrow down the location at a country level. Ofcourse, in both cases if the user is using VPN, you would only records the VPN location. 2. The problem we want to avoid is same IP with same hashing algorithm will produce same value. Which can then be used to co-relate user activity across days. Second, depending on what hashing scheme one is using, rainbow tables can be used to get back the original value. Therefore, to avoid, we used the approach of daily key. Now that I write, we could also use some hash + daily_salt. This should give the same guarantees. - Needs to be checked. 3. In this setup we are not using Cloudwatch but Cloudfront, which is the CDN provider from AWS.
- marcus_holmes 7y ago
  1. Ah, OK, that's useful. I was thinking that dropping the last byte of the IP address would still be able to give us the country via database lookup, but anonymise enough. But getting it via CF (or similar) would be a good alternative. 2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant). But adding a daily salt would decrease the odds, for sure. 3. Sorry, my bad, I meant Cloudfront, but it's Sunday night here and there has been wine ;) Same question: how has Cloud{{thing}} worked out for you? Is it worth the extra complexity?
  - FabHK 7y ago
    > 2. Chances of a hash collision on IP address is pretty small (i.e. statistically insignificant) The problem, as I understand GP, is not so much the fear that two different IPs might collide. The problem is that seeing the same hash, you know that it refers to the same IP, and as such you can correlate users/sites over time (a privacy invasion).
    - marcus_holmes 7y ago
      That makes sense, kinda. Though I'm a little unsure why correlating http requests over an hour is not invading privacy, but doing the same over a week is?