5 ms·
I wrote a crawler a few years ago. I fired it up recently but had little luck in fetching pages. It looked like cloudflare was protecting the site from me. A
by mrkeen 4y ago
I wrote a crawler a few years ago. I fired it up recently but had little luck in fetching pages. It looked like cloudflare was protecting the site from me.
Are any of you other HNers finding the web increasingly difficult to scrape from?
- adamredwoods 4y agoIs it a 301 that keeps looping? I noticed my company's website does that when I try to cURL, and I wondered if it was cookie based, or how to get around it. (EDIT: Yes, I figured it out. To get around the 301 loop. cURL needs to save cookies)
- bjord 4y agoIf you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration (for me, anyway) was pretty easy. https://github.com/venomous/cloudscraper https://github.com/venomous/cloudscraper