7 ms·
I recently undertook my first scraping project, and after trying a number of things landed upon Scrapy. It’s been a blessing. Not only can it handle difficult
by FL33TW00D 6y ago
I recently undertook my first scraping project, and after trying a number of things landed upon Scrapy.
It’s been a blessing. Not only can it handle difficult sites, but it’s super quick to write another spider for the easy sites that provide the JSON blob in a handy single API call.
Only problem I had was getting around cloudflare, tried a few things like puppeteer but no luck.
- gg_no_re 6y agoCheck out https://github.com/clemfromspace/scrapy-cloudflare-middleware https://github.com/clemfromspace/scrapy-cloudflare-middlewar.... I've been running it in production for over a year without any hiccups.
- FL33TW00D 6y agoIs this still working for you? cfscrape that it’s based off looks to be failing due to cloudflare updates