I love scrapy! It’s a wonderful tool. One of the most underrated features is the request caching. It really helps with the problem of finding out your spider c

I love scrapy! It’s a wonderful tool. One of the most underrated features is the request caching. It really helps with the problem of finding out your spider crashed or you didn’t parse all the data you wanted and rerunning the job. Rather than making hundred or thousands of requests you can get them from the cache. One nitpick is that the documentation could be a bit better about integrating scrapy with other Python projects / code rather than running it directly from the command line. Also, some of their internal names are a bit vague. There’s a Spider and a Crawler. What’s the difference? To most people these would be the same thing. This makes reading the source code a little tricky.