1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily c

1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily cost enough in extra server fees to cause an "undue burden". At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content. 2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.