2 ms·
I deal with security and web site performance. If you can do for scraping what Akismet has done for spam on WP, then you have a great product. Aside from e-sh
by jeffatrackaid 12y ago
I deal with security and web site performance. If you can do for scraping what Akismet has done for spam on WP, then you have a great product.
Aside from e-shops, forums (VB, PHPBB, etc) also have a lot of scraping for forum meta-sites and cross-posting.
- aruggirello 12y agomy project employs several techniques to flag and block scrapers; note that any technique will eventually fail against the 'perfect storm': a distributed scraping network featuring camouflaged user agents operating from innocent-looking IPs (from the same ISPs of your regular users) and performing a single, RANDOM request each. There is little one could do to prevent scraping in this scenario; fortunately enough, scraping bot developers don't usually take all those precautions, so identifying them is often much, much easier (some even proudly identify themselves in the user agent string... :)