2 ms·
How do you stop people from scraping your site? Databrokers frequently will mine social sites to build profiles on people. There's legitimate reasons to
by adrr 11y ago
How do you stop people from scraping your site? Databrokers frequently will mine social sites to build profiles on people. There's legitimate reasons to block TOR for read-only content.
- zzzcpan 11y agoScrapers visit many unique URLs. Such request patterns should be fairly easily distinguishable from the rest of the visitors. To do it on a large scale in realtime something like loglog[1] counter could be used. [1] https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
- eastdakota 11y agoHow do you id the scraper from request to request? Cookie? Bots can just discard. IP? Now we're back to the same problem. Some sort of super-cookie browser fingerprinting? Sure, maybe possible, but it's super sketchy and sacrifices anonymity which is important.
- zzzcpan 11y agoI don't think it is necessary to identify the scraper. Something simple should work, like counting how many unique URLs for a particular website got accessed in some period of time and comparing that to similar thing, but counted for known non-bot users (normalized). If one is much bigger than the other one - start requiring CAPTCHAs, but only for those requests, that are not part of known non-bot users set of our probabilistic counter of requests or even a separate bloom filter, depending on what is going to be more efficient and more accurate.
- fabulist 11y agoIt is 2016. The web is public. People scrape sites. Don't show them things they shouldn't see. Don't pretend you're going to stop 80legs from scraping you by blocking Tor.