I think as a scraper, we should generally respect rate-limits. Generally the issue is that it's hard to enforce that against bad users. The few times I've writ

I think as a scraper, we should generally respect rate-limits. Generally the issue is that it's hard to enforce that against bad users. The few times I've written scrapers, one of the things I've done is put contact information in the user-agent so that if there is an issue, the site admins can reach out. So far that's worked out ok for me.