4 ms·
Can you please give some details about your second point? What’s wrong with beautifulsoup?
by sintezcs 5y ago
Can you please give some details about your second point? What’s wrong with beautifulsoup?
- estebarb 5y agoIt is very slow. But personally, I prefer to write my crawlers in Go (custom code, not Colly).
- zatarc 5y agoTry Parsel: https://github.com/scrapy/parsel https://github.com/scrapy/parsel It's way faster and has better support for CSS selectors.
- moehm 5y agoWhat's your problem with Colly? [0] [0] http://go-colly.org/ http://go-colly.org/
- estebarb 5y agoMostly that I started my crawler before learning about Colly and it didn't make sense to rewrite the code. By "not Colly" I just wanted to remark that in Go is relatively easy to write a crawler from scratch.
- thegeekpirate 5y ago> But personally, I prefer to write my crawlers in Go (custom code, not Colly). This is my current setup as well, been scraping on and off for 20+ years now.
- aynyc 5y agoUsing CSS & XPATH to select elements is very natural to web pages. BS4 has very limited CSS selector support and zero XPATH support.