4 ms·
Why is this getting downvoted? I'm also interested in why this approach isn't taken.
by deogeo 7y ago
Why is this getting downvoted? I'm also interested in why this approach isn't taken.
- jhgg 7y agoChrome does something like this, you can learn more about it here: https://codereview.chromium.org/6286072/ https://codereview.chromium.org/6286072/
- mokus 7y agoA couple possibilities I can think of: * the list may be prohibitively large * it exposes to the bad actors exactly which of their scams is detected, so they can simply refine their methods until their sites don’t make “the list”
- Gaelan 7y agoWouldn’t it be pretty easy for the bad actors to check the database anyway? I can’t imagine they would need to query often enough to hit any rate limits.
- cryptonector 7y agoBloom filters take care of the first. There will always be an arms race between attack and defense, so I'm not concerned about the second issue.
- yorwba 7y agoBloom filters can give false positives, and to eliminate them, you'd need to send "data (hashed, anonymized, truncated, or otherwise)" to some entity that has the full list. That's exactly how Google's Safe Browsing API works.
- deogeo 7y agoCan't the bad actors already check each of their sites individually by pretending to be a normal user?
- souterrain 7y agoBad actors can also occasionally poll the safebrowsing API.
- pmoriarty 7y agoExactly how large is it? OS vendors already make a habit of regularly sending gigantic OS updates. I'd have a hard time believing that a compressed list of malware URLs would be noticeably bigger, by comparison. Also, once the list is sent the first time (or just included with the OS so it'd be already present on your device when you bought it), they could just send the deltas as the list changed, and those deltas (especially once compressed) should be relatively small even compared to the original (probably not that large) list.
- NetMageSCW 7y agoGoogle Safe Browsing transparency report lists 40k new bad URLs per week - how large do you think the list is now? It is far, far too large for local processing but k-anonymity is perfectly trustworthy when used with cryptographic hashing.
- cryptonector 7y agoDownvoters who think that might be too much data don't know about Bloom filters.
- morelisp 7y agoPeople who think they know about Bloom filters should consider what attack vectors a false positive with such would allow. (The end result of the thought experiment will be basically what Google does now.)
- robocat 7y agoBloom filters are likely useless in this situation - following facts for phishing only: 1. Phishing sites have a lifecycle of about 15 hours. 2. Most malicious links are hidden within benign domains. 3. About 400,000 phishing sites are created each month. From: https://www.itgovernance.co.uk/blog/4-eye-opening-facts-about-phishing https://www.itgovernance.co.uk/blog/4-eye-opening-facts-abou... I haven't run the numbers, but I am guessing that a clientside solution would have a lot of bandwidth sucking and avoiding false positives is very important. Also with a clientside solution, how are new phishing URLs detected? PS: perhaps try to assume HNers know what a Bloom filter is (I've seen them come up lots of times in comments).
- pmoriarty 7y ago400,000 sounds like a lot, but I wonder how many new URLs Tencent adds to its database each month. I expect they don't add every phishing URL but some small subset of them (possibly even a very small subset.. we'll proably never know). But let's say it is 400,000. I took the URL you linked and made a file of 400,000 copies of it. The file size was 28 MB. I didn't bother compressing that particular file since the URL is the same in each instance, but I expect a file full of actual phishing URLs would probably compress pretty well, so it would probably be significantly less than 28 MB. Considering that OS vendors regularly ship multi-gigabyte size updates, having to download less than 28 MB extra every month shouldn't even be noticeable. If updates needed to be done more frequently, the client could subscribe to get regular updates as they become available.
- lostlogin 7y agoLocal handling seems more Appley too.