4 ms·
I used scrapy a lot. Just my opinion: 1. Instead of creating a urls global variable, use start_requests function. 2. Don't use beautifulsoup to parse, use CSS
by aynyc 5y ago
I used scrapy a lot. Just my opinion:
1. Instead of creating a urls global variable, use start_requests function.
2. Don't use beautifulsoup to parse, use CSS or XPATH.
3. If you are going into multiple pages over and over again, use CrawlSpider with Rule.
- sintezcs 5y agoCan you please give some details about your second point? What’s wrong with beautifulsoup?
- estebarb 5y agoIt is very slow. But personally, I prefer to write my crawlers in Go (custom code, not Colly).
- zatarc 5y agoTry Parsel: https://github.com/scrapy/parsel https://github.com/scrapy/parsel It's way faster and has better support for CSS selectors.
- moehm 5y agoWhat's your problem with Colly? [0] [0] http://go-colly.org/ http://go-colly.org/
- estebarb 5y agoMostly that I started my crawler before learning about Colly and it didn't make sense to rewrite the code. By "not Colly" I just wanted to remark that in Go is relatively easy to write a crawler from scratch.
- thegeekpirate 5y ago> But personally, I prefer to write my crawlers in Go (custom code, not Colly). This is my current setup as well, been scraping on and off for 20+ years now.
- aynyc 5y agoUsing CSS & XPATH to select elements is very natural to web pages. BS4 has very limited CSS selector support and zero XPATH support.