3 ms·

I assumed every browser did this as a part of their safe browsing features. Or is this different somehow?

by josho 7y ago

antoineMoPa 7y ago
Following tweets explain that other browsers send an hashed URL instead. (but I guess it's easy to associate the hash with actual URLs)
- shakna 7y ago
  Other browsers use a _truncated_ hash. Easy enough to match to a known list of bad sites, but exceptionally difficult to reverse engineer a list of sites the user has visited.
  - ChrisLomont 7y ago
    Those hashes have to be essentially unique to prevent blocking good sites erroneous;y, and as such are easily associated to actual sites when you’re the scale of MS or Google.
    - simias 7y ago
      Not if there's an other layer of verification locally. Then you can allow a few false positives with the hash on the remote and they'll get discarded when the full hash is compared locally. That being said 32bits of information seems like quite a lot and can probably used to match the original URL fairly accurately.
      - liability 7y ago
        You also need to consider the greater context. Suppose my history shows I sent a hash that could be one of 100 different known URLs. One of them is about model airplanes, but there is no way to know I picked that one. Chance I like model airplanes? All else being equal.. call it 1% But what if over the next hour I continue to send hashes with random subject matter sets that include model airplanes each time. Suppose the intersection of the possible subject matters shrinks with each additional hash, quite possibly contains only model airplanes. I visited 100 URLs in one hour. For each URL there are 100 others with the same hash but independent topics. Lets say there are 10,000 known topics, but every hash I sent has model airplanes associated with it. Now what are the chances I like model planes? It seems clear to me that this scheme, with logging, reveals a lot in theory. But maybe solving this problem in practice would cost more that the data is worth. For now.
        tialaramex 7y ago
        You skipped a factor, in Safe Browsing Update API (the mode actual web browsers like Firefox use) that step where you send a 4 byte prefix only happens when the prefix already matched a local hotlist. So if you visit 100 URLs in an hour, and using the local hotlist allows your browser to discard all but 1 of those URLs as definitely not bad, that's only one URL prefix checked against Google, not 100 so there's no triangulation. And the actual number isn't 1-in-100, that's why they picked 4 byte prefixes. I haven't actually checked, but by eyeball from playing with this data for work I would guess 1-in-a-million. So, you visit 100 URLs about model planes, one of them happens to be a 1-in-a-million match to a possible badware site, your browser sends the 4-byte hash prefix to Google, it gives back the 4-byte prefix of the badware site it was worried about which is different, your browser goes "Phew, good" and nothing happens. There's just not really an opportunity for tracking here.
        tialaramex 7y ago
        Ugh, editing goof - that last bit of my explanation is wrong, Google sends the whole SHA256 hash of the bad site's URL and your browser compares -that- to the full hash it calculates for the model plane site. The four byte prefix will be the same of course, if it wasn't we wouldn't get this far.
    - hope-striker 7y ago
      Here's what all the browsers do (Firefox, Google and Safari all use Google's Safe Browsing service). 1. The truncated hash is checked against local database. 2. If there's a hit, send the truncated hash to Google, and receive a list of full hashes for the truncated hashes. 3. Check if the full hash is in that list. If so, warn. I'm not certain that the hash is truncated enough, but the general idea is a pretty straightforward trade-off between the amount of information Google needs and the amount of information your computer needs.
- eat_veggies 7y ago
  The browser sends a prefix (first few bytes) of the hash [1], and gets back a list of full hashes to compare locally. This is called k-anonymity, and it's good enough for even services like Have I Been Pwned to use it for password lookups [2] [1] https://www.chromium.org/developers/design-documents/safebrowsing https://www.chromium.org/developers/design-documents/safebro... [2] https://blog.cloudflare.com/validating-leaked-passwords-with-k-anonymity/ https://blog.cloudflare.com/validating-leaked-passwords-with...

I assumed every browser did this as a part of their safe browsing features. Or is this different somehow?

by josho 7y ago

I assumed every browser did this as a part of their safe browsing features. Or is this different somehow?

Following tweets explain that other browsers send an hashed URL instead. (but I guess it's easy to associate the hash with actual URLs)

shakna 7y ago
Other browsers use a _truncated_ hash. Easy enough to match to a known list of bad sites, but exceptionally difficult to reverse engineer a list of sites the user has visited.
- ChrisLomont 7y ago
  Those hashes have to be essentially unique to prevent blocking good sites erroneous;y, and as such are easily associated to actual sites when you’re the scale of MS or Google.
  - simias 7y ago
    Not if there's an other layer of verification locally. Then you can allow a few false positives with the hash on the remote and they'll get discarded when the full hash is compared locally. That being said 32bits of information seems like quite a lot and can probably used to match the original URL fairly accurately.
    - liability 7y ago
      You also need to consider the greater context. Suppose my history shows I sent a hash that could be one of 100 different known URLs. One of them is about model airplanes, but there is no way to know I picked that one. Chance I like model airplanes? All else being equal.. call it 1% But what if over the next hour I continue to send hashes with random subject matter sets that include model airplanes each time. Suppose the intersection of the possible subject matters shrinks with each additional hash, quite possibly contains only model airplanes. I visited 100 URLs in one hour. For each URL there are 100 others with the same hash but independent topics. Lets say there are 10,000 known topics, but every hash I sent has model airplanes associated with it. Now what are the chances I like model planes? It seems clear to me that this scheme, with logging, reveals a lot in theory. But maybe solving this problem in practice would cost more that the data is worth. For now.
      - tialaramex 7y ago
        You skipped a factor, in Safe Browsing Update API (the mode actual web browsers like Firefox use) that step where you send a 4 byte prefix only happens when the prefix already matched a local hotlist. So if you visit 100 URLs in an hour, and using the local hotlist allows your browser to discard all but 1 of those URLs as definitely not bad, that's only one URL prefix checked against Google, not 100 so there's no triangulation. And the actual number isn't 1-in-100, that's why they picked 4 byte prefixes. I haven't actually checked, but by eyeball from playing with this data for work I would guess 1-in-a-million. So, you visit 100 URLs about model planes, one of them happens to be a 1-in-a-million match to a possible badware site, your browser sends the 4-byte hash prefix to Google, it gives back the 4-byte prefix of the badware site it was worried about which is different, your browser goes "Phew, good" and nothing happens. There's just not really an opportunity for tracking here.
        tialaramex 7y ago
        Ugh, editing goof - that last bit of my explanation is wrong, Google sends the whole SHA256 hash of the bad site's URL and your browser compares -that- to the full hash it calculates for the model plane site. The four byte prefix will be the same of course, if it wasn't we wouldn't get this far.
  - hope-striker 7y ago
    Here's what all the browsers do (Firefox, Google and Safari all use Google's Safe Browsing service). 1. The truncated hash is checked against local database. 2. If there's a hit, send the truncated hash to Google, and receive a list of full hashes for the truncated hashes. 3. Check if the full hash is in that list. If so, warn. I'm not certain that the hash is truncated enough, but the general idea is a pretty straightforward trade-off between the amount of information Google needs and the amount of information your computer needs.
eat_veggies 7y ago
The browser sends a prefix (first few bytes) of the hash [1], and gets back a list of full hashes to compare locally. This is called k-anonymity, and it's good enough for even services like Have I Been Pwned to use it for password lookups [2] [1] https://www.chromium.org/developers/design-documents/safebrowsing https://www.chromium.org/developers/design-documents/safebro... [2] https://blog.cloudflare.com/validating-leaked-passwords-with-k-anonymity/ https://blog.cloudflare.com/validating-leaked-passwords-with...