5 ms·

They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single

They would definitely have a list of the domains but a hash isn't just generated from the domain root. They would have to have a hash generated for every single URL and then they would have to collect every single instance where a hash came up as a "hit" for that hash. They'd then have to model these out to determine which hashes were being hit that were related from a specific IP address which, in my opinion, would be very, very computationally expensive.

lallysingh 7y ago
A few permutations of each would do it. http://foo.com http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.
- dpkonofa 7y ago
  That doesn't make sense. http://foo.com/index.html http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix. You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.
  - r00fus 7y ago
    But wouldn’t favicon.ico be called from any visit to any of the subpages off the domain’s roots?
  - lallysingh 7y ago
    Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly. Edit: example: P(baduser | hashA )= ~0 P(baduser | hashA and hashB and ... hashF) = 0.75 One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.
  - deleted 7y ago
    [deleted]
  - kccqzy 7y ago
    The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).
  - stordoff 7y ago
    I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites. Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.
    - dpkonofa 7y ago
      While that might be true, this feels like it would be a long way off. The prefixes could potentially match millions of sites and the results are only then matched on the client side. The server would never know which prefixes are the “actual” hit and which are obfuscated prefixes for completely different domains.
      - lallysingh 7y ago
        Simple: the service provider visits these sites itself, gets hashes of all the resources for the homepage and relevant links (or just spiders them). With the set of hashes, look at prefixes. Users who request one prefix in that set aren't suspicious. Users with many prefixes in that set are.
  - usrusr 7y ago
    This form of surveillance would be a game of bingo: for a given "bad" domain you have multiple individual URLs that a browser would hit. Put a few of them on the trigger watchlist, grouped by domain, and when a browser hits some threshold number of the same group within seconds the algorithm shouts "bingo" because chances are this won't be unrelated collisions. Note also that mass surveillance isn't partucularly concerned about false positives.
- zwaps 7y ago
  It's literally enough that the system can match ~some~ users to just one exact URL. That's a working system, you will see.
Scirra_Tom 7y ago
If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.

lallysingh 7y ago

A few permutations of each would do it. http://foo.com http://foo.com, with /index.html, /favicon.ico, etc. You don't have to match all of them to mark a user (who in China is 1:1 to IP, IIRC) as visitor of undesirable site.

dpkonofa 7y ago
That doesn't make sense. http://foo.com/index.html http://foo.com/index.html would hash to something completely different from http://foo.com/favicon.ico http://foo.com/favicon.ico. On top of that, the prefix, which is only the first 32 or 48 bits, would match for thousands of other URLs whose total hash would be completely different. Additionally, hash collisions are a thing so 2 completely different inputs could potentially return the same full hash, much less just the prefix. You would need a very, very, very large database to keep track of all the permutations necessary to determine a domain name.
- r00fus 7y ago
  But wouldn’t favicon.ico be called from any visit to any of the subpages off the domain’s roots?
- lallysingh 7y ago
  Or the hashes of all the resources loaded by those pages. That collection of prefixes is going to start getting unique quickly. Edit: example: P(baduser | hashA )= ~0 P(baduser | hashA and hashB and ... hashF) = 0.75 One prefix doesn't do it, but a bunch together in a smallish time period can fingerprint a site quite well. This is analagous to browser fingerprinting.
- deleted 7y ago
  [deleted]
- kccqzy 7y ago
  The safe browsing hashing algorithm takes that into account. You are supposed to canonicalize and simplify the URL. For example https://example.com/some/path?k=w#hash https://example.com/some/path?k=w#hash would have the hash removed, query parameter optionally removed, then path components optionally removed, and finally sub domains removed up to the public suffix plus one, and produce multiple hashes (up to 30 of them).
- stordoff 7y ago
  I think OP means that the collection of visited prefixes can be indicative. If foo.example.com, bar.example.com, and baz.example.com are "undesirable" sites, and you have requested hash prefixes matching those for foo.example.com/index, foo.example.com/favicon, bar.example.com/index, bar.example.com/favico, baz.example.com/index, and baz.example.com/favicon, you might be able to start making educated guesses that the user is visiting such sites. It really depends how much of an overlap for "acceptable" browsing falling into these buckets there is, or if you can weight certain prefixes as likely to be indicative of visiting "undesirable" sites. You might not able to say this user visited this specific page at this specific time, but you might be able to fingerprint a user as visiting a particular category of sites. Furthermore, I'm unclear how exactly Safe Browsing is implemented, but if it checks every resource on a page, the set of prefixes for all resources on a given page may leak more data than one query on its own, so you can look for requests for that set of queries in a short time frame to identify a particular page.
  - dpkonofa 7y ago
    While that might be true, this feels like it would be a long way off. The prefixes could potentially match millions of sites and the results are only then matched on the client side. The server would never know which prefixes are the “actual” hit and which are obfuscated prefixes for completely different domains.
    - lallysingh 7y ago
      Simple: the service provider visits these sites itself, gets hashes of all the resources for the homepage and relevant links (or just spiders them). With the set of hashes, look at prefixes. Users who request one prefix in that set aren't suspicious. Users with many prefixes in that set are.
- usrusr 7y ago
  This form of surveillance would be a game of bingo: for a given "bad" domain you have multiple individual URLs that a browser would hit. Put a few of them on the trigger watchlist, grouped by domain, and when a browser hits some threshold number of the same group within seconds the algorithm shouts "bingo" because chances are this won't be unrelated collisions. Note also that mass surveillance isn't partucularly concerned about false positives.
zwaps 7y ago
It's literally enough that the system can match ~some~ users to just one exact URL. That's a working system, you will see.

Scirra_Tom 7y ago

If I understand correctly, the full URL is hashed (or at least the first n bits of the full URL). It really isn't computationally expensive to work out undesirable URL's being visited because the undesirable domains are known, and it would be trivial for them to periodically crawl the undesirable domains for all known internal URLs and build a dictionary of hashes.