For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that th

For non-commercial projects I always use at least a one second delay, even for large sites. For commercial projects I have actually advocated to clients that they contact the site's owners and ask to arrange a data use agreement. And a couple have really done it and been satisfied by the result which was cleaner data and continuous updates at a reasonable cost. Seriously, why not? I've never lost anything doing this, you can always scrape (with delay) as Plan B. A little dose of ethics goes a long way!