Detect: Look for high frequency access with low variability in the delay between same-type resource requests. It's a relatively simple heuristic, but great pre

Detect: Look for high frequency access with low variability in the delay between same-type resource requests. It's a relatively simple heuristic, but great precision & recall for classifying scrapers vs. manual access. Block: You can't block a sufficiently determined expert scraper. You can filter out a lot of scraping if that's a meaningful goal, but it's often not. A good strategy here is rate limiting + header inspection + CAPTCHA on high activity. Possibly add transfer quotas if you have a lot of large assets which might cause problems at low request rates. This will deal with "script kiddies" and automated tools which account for the majority of cases where scraping could potentially impact the server's ability to respond to other users. It won't block researchers or other scrapers with more sophisticated skills. I've seen people go wild here, to the point of silly things like displaying database record text as images. Complicated schemes generally impact UX for your routine users, while making it take slightly more time to write a scraper that's already "build once, use until the site is redesigned."