2 ms·
How do you id the scraper from request to request? Cookie? Bots can just discard. IP? Now we're back to the same problem. Some sort of super-cookie browser fing
by eastdakota 11y ago
How do you id the scraper from request to request? Cookie? Bots can just discard. IP? Now we're back to the same problem. Some sort of super-cookie browser fingerprinting? Sure, maybe possible, but it's super sketchy and sacrifices anonymity which is important.
- zzzcpan 11y agoI don't think it is necessary to identify the scraper. Something simple should work, like counting how many unique URLs for a particular website got accessed in some period of time and comparing that to similar thing, but counted for known non-bot users (normalized). If one is much bigger than the other one - start requiring CAPTCHAs, but only for those requests, that are not part of known non-bot users set of our probabilistic counter of requests or even a separate bloom filter, depending on what is going to be more efficient and more accurate.