4 ms·

I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get

I have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond looking at logs and doing IP bans?

pjc50 9y ago
The cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.
jstarfish 9y ago
IPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username. Establish a baseline as to what constitutes "normal" browsing. Any user exceeding a threshold of page requests gets rate limited/banned. Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.
amelius 9y ago
Captchas are one (drastic) option.
- nojvek 9y ago
  Any image based captcha where one needs to identify words from an image can be easily broken by algorithms now. Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.
mmcwilliams 9y ago
I'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0]. IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble. [0] https://news.ycombinator.com/item?id=16868012 https://news.ycombinator.com/item?id=16868012

pjc50 9y ago

The cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.

jstarfish 9y ago

IPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username. Establish a baseline as to what constitutes "normal" browsing. Any user exceeding a threshold of page requests gets rate limited/banned. Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.

amelius 9y ago

Captchas are one (drastic) option.

nojvek 9y ago
Any image based captcha where one needs to identify words from an image can be easily broken by algorithms now. Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.

mmcwilliams 9y ago

I'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0]. IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble. [0] https://news.ycombinator.com/item?id=16868012 https://news.ycombinator.com/item?id=16868012