4 ms·

I don't particularly have a problem with scraping, but I could really do without whoever keeps trying to scrape lots of pages very quickly from hundreds of IPs

• vorpalhex 6y ago
  I think as a scraper, we should generally respect rate-limits. Generally the issue is that it's hard to enforce that against bad users. The few times I've written scrapers, one of the things I've done is put contact information in the user-agent so that if there is an issue, the site admins can reach out. So far that's worked out ok for me.