5 ms·
A guide to preventing Webscraping
- ttsda 6y agoIf your data is reasonably valuable, it will be impossible to deter a minimally invested developer armed with puppeteer (with some modifications), residential proxy services, and captcha solving (way less than a cent per captcha). Most sites that attempt to do it hinder their users more than they do the scrapers.
- wolrah 6y agoAgreed 100%. Basically anything that would stop a scraper from downloading or interpreting a single page will also get in the way of search engines and screen readers. The former is generally not desirable and the latter may actually be illegal for some organizations. Just give up if this is your goal. Rate limiting and other approaches to restrict bulk downloading of your entire dataset are more practical, but are still generally easy to work around if one is sufficiently determined unless you require authentication for all access.
- wow_yes 6y agoWhy would I want to do that though? This is deeply against the spirit of the web
- cute_boi 6y agothat guide teaches to become too aggressive which is unnecessary for many case. But in turn you are damaging user ergonomics. If google start showing captcha on every search yes they protected it from bot but is that what normal real user want? No. If you data is that much valuable offer them at price so bot shouldn't be issue in many case? But these days random eCommerce website asking for captcha is totally ridiculous. Some alibaba website are so frustrating that i don't even browse them anymore.
- user5994461 6y ago>>> Check the Referer header Warning on this. Many browsers always leave the referer empty nowadays. A user with no referer is very likely to be a human.
- stevenalowe 6y agoone solution omitted: provide an API so scraping is unnecessary
- chlopez21 6y agoThanks, this is really useful!
- deleted 6y ago[deleted]