3 ms·

Does anyone have tips for polite scraping? Just making sure not to hurt a site (or get IP blocked)?

by jjice 6y ago

Nextgrid 6y ago
Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do it from a single IP. Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach). Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.
abdusco 6y ago
Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I tweak these values to what I estimate the website could handle (depending on how large it is). The page also outlines how throttling algorithm works, which you use as a starting point for your own work. https://docs.scrapy.org/en/latest/topics/autothrottle.html https://docs.scrapy.org/en/latest/topics/autothrottle.html
dgb23 6y ago
I assume that there is merit in throttling the request frequency.
andyfleming 6y ago
This is a tough problem because if you asked most sites/admins if you could scrape their site, they don’t really have much motivation to say yes. For some it feels like you are stealing their hard work. For others they don’t want to pay for the requests your scraper will make to their site, etc. Regardless, I’m an ideal world, some polite things would be to: - ask for permission before scraping, explaining how it’s neutral or positive for their business (so they are more likely to support your continued scraping or even potentially provide a more cost-effective format/API for you) - scrape at a reasonable pace - scrape off hours - make requests with a clear user agent so they know where the requests are coming from
heavyset_go 6y ago
Generous caching and throttling.
II2II 6y ago
Something that I rarely see mentioned: only scrape what you need. If the site provides some means of limiting what it returns to you (a particular path, category, or search), take advantage of that. If you know that the information is on a limited number of pages and you only need to acquire it once, there it is perfectly fine to download those pages manually then have your program extract the required information automatically. (That may even save you from some work.)

Does anyone have tips for polite scraping? Just making sure not to hurt a site (or get IP blocked)?

by jjice 6y ago

Does anyone have tips for polite scraping? Just making sure not to hurt a site (or get IP blocked)?

Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do it from a single IP. Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach). Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.

abdusco 6y ago

Scrapy (a popular python scraping framework) has an auto-throttling feature that lets you limit concurrent requests and requests per minute for a domain. I tweak these values to what I estimate the website could handle (depending on how large it is). The page also outlines how throttling algorithm works, which you use as a starting point for your own work. https://docs.scrapy.org/en/latest/topics/autothrottle.html https://docs.scrapy.org/en/latest/topics/autothrottle.html

dgb23 6y ago

I assume that there is merit in throttling the request frequency.

andyfleming 6y ago

This is a tough problem because if you asked most sites/admins if you could scrape their site, they don’t really have much motivation to say yes. For some it feels like you are stealing their hard work. For others they don’t want to pay for the requests your scraper will make to their site, etc. Regardless, I’m an ideal world, some polite things would be to: - ask for permission before scraping, explaining how it’s neutral or positive for their business (so they are more likely to support your continued scraping or even potentially provide a more cost-effective format/API for you) - scrape at a reasonable pace - scrape off hours - make requests with a clear user agent so they know where the requests are coming from

heavyset_go 6y ago

Generous caching and throttling.

II2II 6y ago

Something that I rarely see mentioned: only scrape what you need. If the site provides some means of limiting what it returns to you (a particular path, category, or search), take advantage of that. If you know that the information is on a limited number of pages and you only need to acquire it once, there it is perfectly fine to download those pages manually then have your program extract the required information automatically. (That may even save you from some work.)