I've gone through this process twice- one about six months ago, and once just this week. In the first event the content wasn't clearly licensed and the site as

I've gone through this process twice- one about six months ago, and once just this week. In the first event the content wasn't clearly licensed and the site as somewhat small, so I didn't want to break them. I emailed them and they gave us permission but only if we only crawled one page per ten seconds. Took us a weekend, but we got all the data and did so in a way that respected their site. The second one was this last week and was part of a personal project. All of the content was over an open license (creative commons), and the site was hosted on a platform that can take a ton of traffic. For this one I made sure we weren't hitting it too hard (scrapy has some great autothrottle options), but otherwise didn't worry about it too much. Since the second project is personal I open sourced the crawler if you're curious- https://github.com/tedivm/scp_crawler https://github.com/tedivm/scp_crawler