2 ms·
> [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website; Two things: 1) The Eng
by traceroute66 5y ago
> [Disrupt] the services by placing an undue burden on a Project website or the networks or servers connected with a Project website;
Two things:
1) The English wikipedia *alone* gets 250 million page views per day ! So you would have to be doing an awful lot to cause "undue burden".
2) The Wikipedia robots.txt page openly implies that crawling (and therefore scraping) is acceptable *as long as* you do it in a rate-limited fashion, e.g.:
>Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please.
> There are a lot of pages on this site, and there are some misbehaved spiders out there that go _way_ too fast.
>Sorry, wget in its recursive mode is a frequent problem. Please read the man page and use it properly; there is a --wait option you can use to set the delay between hits, for instance.
- learc83 5y ago1. You'd be surprised what kind of traffic scrapers can generate. I've seen scraping companies employing botnets to get around rate limiting that could easily cost enough in extra server fees to cause an "undue burden". At a previous company we had the exact problem that we published all of our content as machine readable xml, but we had scrapers costing us money by insisting on using our search interface to access our content. 2. No one is going to jail for scraping a few thousand or even a few million pages, but just because low speed web crawlers are allowed to index the site, doesn't mean scraping for every possible use is permitted.