3 ms·
Most really don't. A lot of big sites don't seem to care, at least in my experience. The few that I've seen just 'ban' your IP for a few minutes. If you hit Wi
by madamelic 10y ago
Most really don't. A lot of big sites don't seem to care, at least in my experience.
The few that I've seen just 'ban' your IP for a few minutes. If you hit Wikipedia too much too quickly, they will essentially refuse to serve you for a while. It was a number of years ago I was doing it, but basically you would be scraping then you would just stop getting info (Maybe I wasn't reading response codes and could've realized quicker what was happening)
- detaro 10y agoWikipedia provides you with an API and guidelines on how to use it, so you really shouldn't be scraping it directly or so much you hit enforced limits.
- mod 10y ago"It was a number of years ago..."
- user5994461 10y agoWikipedia provides archives of all its content. No need to scrap it when you can readily download a nicely formatted .xml.zip file containing all knowledge written by mankind.