3 ms·

I've used scrapy a few times and it's never felt like a big over-arching framework. I've been able to change what I need, and what it does for me are all the t

by IanCal 10y ago

I've used scrapy a few times and it's never felt like a big over-arching framework. I've been able to change what I need, and what it does for me are all the things I'd have to have done myself (caching, parallel requests but throttling per domain, dropping into debug mode, scheduled runs, etc). Really, it feels more like a skeleton + lots of sensible defaults. The meat of the code will be in the parsing, and so if for some reason I really needed to move away from it then I wouldn't feel particularly tied to it.

brilliantcode 10y ago
Interesting. So the web crawling/page fetching component is a major value add to you as a developer? Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? What about changing the parser and crawler as the websites changes? What other pain points about scrapy do you have?
- IanCal 10y ago
  > Interesting. So the web crawling/page fetching component is a major value add to you as a developer? When I have a scraping task, yes. It's a set of things that are required each time but also a bit fiddly to get right, and scrapy has solved them well. > Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? These I see as fundamentally having to be things I code as they're the parts that are different each time. > What about changing the parser and crawler as the websites changes? Really just a cost of doing scraping. Anything fancier has generally taken more time and created more problems than just assuming an ongoing maintenance cost. The debugging shell in scrapy is very useful for this. Last big job I did I also built a cache that you could query by time, so all versions of the page seen were stored which was very useful for debugging intermittent problems, and finding page changes. I don't know if scrapy has this in its cache, I don't think so but wouldn't conflict with it.

by IanCal 10y ago

brilliantcode 10y ago

Interesting. So the web crawling/page fetching component is a major value add to you as a developer? Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? What about changing the parser and crawler as the websites changes? What other pain points about scrapy do you have?

IanCal 10y ago
> Interesting. So the web crawling/page fetching component is a major value add to you as a developer? When I have a scraping task, yes. It's a set of things that are required each time but also a bit fiddly to get right, and scrapy has solved them well. > Whereas the parsing is less of a value add because you prefer to code the parser yourself so that you have more control? These I see as fundamentally having to be things I code as they're the parts that are different each time. > What about changing the parser and crawler as the websites changes? Really just a cost of doing scraping. Anything fancier has generally taken more time and created more problems than just assuming an ongoing maintenance cost. The debugging shell in scrapy is very useful for this. Last big job I did I also built a cache that you could query by time, so all versions of the page seen were stored which was very useful for debugging intermittent problems, and finding page changes. I don't know if scrapy has this in its cache, I don't think so but wouldn't conflict with it.