4 ms·

Wouldn't tencent, having a list of politically undesirable domains, already know their hashes? Now the browsers that visit them phone tencent to let them know.

Wouldn't tencent, having a list of politically undesirable domains, already know their hashes? Now the browsers that visit them phone tencent to let them know. This is useful when the actual http(s) transfer is under vpn wrapping. Maybe it won't matter anymore.

dpkonofa 7y ago
They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single URL and then they would have to collect every single instance where a hash came up as a "hit" for that hash. They'd then have to model these out to determine which hashes were being hit that were related from a specific IP address which, in my opinion, would be very, very computationally expensive.
- lallysingh 7y ago
  A few permutations of each would do it. http://foo.com http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.
  - dpkonofa 7y ago
    That doesn't make sense. http://foo.com/index.html http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix. You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.
    - r00fus 7y ago
      But wouldn’t favicon.ico be called from any visit to any of the subpages off the domain’s roots?
    - lallysingh 7y ago
      Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly. Edit: example: P(baduser | hashA )= ~0 P(baduser | hashA and hashB and ... hashF) = 0.75 One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.
    - deleted 7y ago
      [deleted]
    - kccqzy 7y ago
      The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).
    - stordoff 7y ago
      I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites. Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.
  - zwaps 7y ago
    It's literally enough that the system can match ~some~ users to just one exact URL. That's a working system, you will see.
- Scirra_Tom 7y ago
  If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.

dpkonofa 7y ago

They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single URL and then they would have to collect every single instance where a hash came up as a "hit" for that hash. They'd then have to model these out to determine which hashes were being hit that were related from a specific IP address which, in my opinion, would be very, very computationally expensive.

lallysingh 7y ago
A few permutations of each would do it. http://foo.com http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.
- dpkonofa 7y ago
  That doesn't make sense. http://foo.com/index.html http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix. You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.
  - r00fus 7y ago
    But wouldn’t favicon.ico be called from any visit to any of the subpages off the domain’s roots?
  - lallysingh 7y ago
    Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly. Edit: example: P(baduser | hashA )= ~0 P(baduser | hashA and hashB and ... hashF) = 0.75 One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.
  - deleted 7y ago
    [deleted]
  - kccqzy 7y ago
    The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).
  - stordoff 7y ago
    I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites. Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.
- zwaps 7y ago
  It's literally enough that the system can match ~some~ users to just one exact URL. That's a working system, you will see.
Scirra_Tom 7y ago
If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.