3 ms·

Interesting. So the web crawling/page fetching component is a major value add to you as a developer? Whereas the parsing is less of a value add because you pre

Interesting. So the web crawling/page fetching component is a major value add to you as a developer? Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? What about changing the parser and crawler as the websites changes? What other pain points about scrapy do you have?

IanCal 10y ago
> Interesting. So the web crawling/page fetching component is a major value add to you as a developer? When I have a scraping task, yes. It's a set of things that are required each time but also a bit fiddly to get right, and scrapy has solved them well. > Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? These I see as fundamentally having to be things I code as they're the parts that are different each time. > What about changing the parser and crawler as the websites changes? Really just a cost of doing scraping. Anything fancier has generally taken more time and created more problems than just assuming an ongoing maintenance cost. The debugging shell in scrapy is very useful for this. Last big job I did I also built a cache that you could query by time, so all versions of the page seen were stored which was very useful for debugging intermittent problems, and finding page changes. I don't know if scrapy has this in its cache, I don't think so but wouldn't conflict with it.