3 ms·

I am a bit skeptical about hashing URLs. Can malware sites just change few parameters and the hash will change right?

by est 7y ago

dannyw 7y ago
The hash is the domain name, I.e. ycombinator.com for https://news.ycombinator.com/reply?id=21254732&goto=item%3Fid%3D21254166%2321254732 https://news.ycombinator.com/reply?id=21254732&goto=item%3Fi...
- dpkonofa 7y ago
  Unless I'm mistaken, that's not true. The hash is computed from the full path and the prefix is the first 32-bits of the hash (or 48-bits if using Safe Browsing).
  - bigiain 7y ago
    Looks right. From your links upthread to Google's Safe Browsing API docs: Canonicalize("http://www.evil.com/blah#frag" http://www.evil.com/blah#frag") = "http://www.evil.com/blah"; http://www.evil.com/blah"; Canonicalize("http://evil.com/foo?bar;" http://evil.com/foo?bar;") = "http://evil.com/foo?bar;"; http://evil.com/foo?bar;"; So fragments get dropped (as expected) buy query params do not (also, in retrospect, what I'd expect to make it work at all...) So https://news.ycombinator.com/reply?id=21254732 https://news.ycombinator.com/reply?id=21254732 will not end up hashing "https://ycombinator.com" https://ycombinator.com", but the whole thing including the path and query string.
tialaramex 7y ago
Safe Browsing has a canonicalization step. Given a URL the Safe Browsing implementation calculates several hashes, each of which is checked. They're basically cutting the URL up into gradually less specific URLs, so e.g. maybe the site malware.example.com is a friendly site discussing Malware, no harm there, but https://malware.example.com/samples/ https://malware.example.com/samples/ is entirely full of sample malware, don't want that, so when you try to go to https://malware.example.com/samples/windowsmacker.exe https://malware.example.com/samples/windowsmacker.exe the browser hashes not only malware.example.com but also malware.example.com/samples and malware.example.com/windowsmacker.exe This lets Google flag the actual problem, without freaking out for users who aren't in harms way. Millions of users visiting the discussion site see nothing, but everybody directed to an actual malware installer gets a warning. So a "prefix" operation happens in the code twice, once in processing the SHA-256 hash to get a 32-bit prefix which is all Google / TenCent are ever shown but also before that during canonicalisation to figure out a list of hashes (not a single hash) for each URL visited.

I am a bit skeptical about hashing URLs. Can malware sites just change few parameters and the hash will change right?

by est 7y ago

I am a bit skeptical about hashing URLs. Can malware sites just change few parameters and the hash will change right?

The hash is the domain name, I.e. ycombinator.com for https://news.ycombinator.com/reply?id=21254732&goto=item%3Fid%3D21254166%2321254732 https://news.ycombinator.com/reply?id=21254732&goto=item%3Fi...

dpkonofa 7y ago
Unless I'm mistaken, that's not true. The hash is computed from the full path and the prefix is the first 32-bits of the hash (or 48-bits if using Safe Browsing).
- bigiain 7y ago
  Looks right. From your links upthread to Google's Safe Browsing API docs: Canonicalize("http://www.evil.com/blah#frag" http://www.evil.com/blah#frag") = "http://www.evil.com/blah"; http://www.evil.com/blah"; Canonicalize("http://evil.com/foo?bar;" http://evil.com/foo?bar;") = "http://evil.com/foo?bar;"; http://evil.com/foo?bar;"; So fragments get dropped (as expected) buy query params do not (also, in retrospect, what I'd expect to make it work at all...) So https://news.ycombinator.com/reply?id=21254732 https://news.ycombinator.com/reply?id=21254732 will not end up hashing "https://ycombinator.com" https://ycombinator.com", but the whole thing including the path and query string.

tialaramex 7y ago

Safe Browsing has a canonicalization step. Given a URL the Safe Browsing implementation calculates several hashes, each of which is checked. They're basically cutting the URL up into gradually less specific URLs, so e.g. maybe the site malware.example.com is a friendly site discussing Malware, no harm there, but https://malware.example.com/samples/ https://malware.example.com/samples/ is entirely full of sample malware, don't want that, so when you try to go to https://malware.example.com/samples/windowsmacker.exe https://malware.example.com/samples/windowsmacker.exe the browser hashes not only malware.example.com but also malware.example.com/samples and malware.example.com/windowsmacker.exe This lets Google flag the actual problem, without freaking out for users who aren't in harms way. Millions of users visiting the discussion site see nothing, but everybody directed to an actual malware installer gets a warning. So a "prefix" operation happens in the code twice, once in processing the SHA-256 hash to get a 32-bit prefix which is all Google / TenCent are ever shown but also before that during canonicalisation to figure out a list of hashes (not a single hash) for each URL visited.