3 ms·
Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do
by Nextgrid 6y ago
Start by assuming good faith from the website regarding scrapers and clearly identify yourself in your user-agent (provide a contact email or something) and do it from a single IP.
Submit a reasonable amount of requests/second and ramp it up slowly depending on the response time and size/popularity of the website (if it's a national e-commerce brand you can hit it with hundreds per second with no expected ill effects because that's a drop in the bucket compared to their overall traffic, but a very small website on shared hosting might warrant a more cautious approach).
Of course, if they are hostile to it and don't like the "being nice" approach then fight fire with fire and try to get what you want as fast as possible while blending into their overall traffic so they can't identify nor block it. It might sometimes even involve paying real people (on Mechanical Turk or similar) to just browse the website as they normally would and scrape it manually.