I tried scrapy many years ago when it was pretty new. I felt like it mostly just got in the way back then and resorted to normal HTTP lib scraping methods like

I tried scrapy many years ago when it was pretty new. I felt like it mostly just got in the way back then and resorted to normal HTTP lib scraping methods like you've described until recently, when I decided to give scrapy another try. I really enjoyed it this time. It has mostly figured out how to be unobtrusive and it now provides a lot of handy stuff out of the box, like the scrapy shell, the ability to easily retry certain error codes, a built-in caching mechanism so that multiple iterative runs are semi-reasonable, and an intelligent crawler that automatically ignores duplicate links from the same session. Compatibility with things like Scrapy Cloud is a nice bonus. It's worth a try if you haven't looked at it in a while.