3 ms·
As someone who used to run a heavily trafficked and heavily scrapped site, some tips from an operator: - Make sure your scrapper has both a reasonable delay (o
by jedberg 7y ago
As someone who used to run a heavily trafficked and heavily scrapped site, some tips from an operator:
- Make sure your scrapper has both a reasonable delay (one request per second or slower) and a proper backoff. It you start getting errors, back off. We never cared about scrapers, until we noticed them, and we only noticed them if they hit us too hard, we told them to back off, and then they didn't.
- Look deep for an API. A ton of people would scrape reddit without realizing we had (an admittedly poorly marketed) API for doing just that.
- Respect robots.txt. That was another way to get noticed quickly -- hitting forbidden URLs. If you hit a forbidden URL too often, you'd start getting 500 errors, and if you didn't back off, you'd get banned from using the site. It was an easy way to tell if someone was not a well behaved scraper.
- BerislavLopac 7y ago> we only noticed them if they hit us too hard, we told them to back off, and then they didn't Response code 429 is your friend: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#429 https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#429
- remram 7y agoYou should back off whatever the error. This is on the client to implement. 429 is not directly supported by HTTP libraries to make the client wait, so I don't feel like it would help getting misbehaving bots to slow down.
- jedberg 7y agoIf you're writing a scraper, you should handle any http error code and back off. And if you want to get really pedantic, 429 didn't exist when we did this. It wasn't approved until April 2012 and the first patches for it didn't show up until around 2014. We could have monkey patched if we really wanted to, but we didn't really want to.