2 ms·
my project employs several techniques to flag and block scrapers; note that any technique will eventually fail against the 'perfect storm': a distributed scrapi
by aruggirello 12y ago
my project employs several techniques to flag and block scrapers; note that any technique will eventually fail against the 'perfect storm': a distributed scraping network featuring camouflaged user agents operating from innocent-looking IPs (from the same ISPs of your regular users) and performing a single, RANDOM request each.
There is little one could do to prevent scraping in this scenario; fortunately enough, scraping bot developers don't usually take all those precautions, so identifying them is often much, much easier (some even proudly identify themselves in the user agent string... :)