It sounds like what you're really asking is this ... There are some websites whose primary business model is providing content in exchange for something: a sub

It sounds like what you're really asking is this ... There are some websites whose primary business model is providing content in exchange for something: a subscription fee, or advertising eyeballs. They have a very strong financial interest in your not scraping their content and providing it to others on different terms. There are other websites who make some content available and explicitly authorize people to use it: various datasets and RSS feeds and such. And then there is a wide swath of websites that have adopted generic TOS that prohibit scraping, or they prohibit it because they haven't given it much thought and can't think of any particular reason off the top of their heads to permit it. So what you really want to know is what sites in the third category would consider a sensible scraping policy, if they had to give it sufficient thought. In other words, if they don't just default to a prohibition because it's already in a TOS template or because they haven't thought it through, what's the rationale for either blocking or not blocking scrapers?