67 ms·
I recently had to scrape the yellow pages for a list of doctors around chicago. My first script caused them to block my ip address. The warning page was cute,
by rauljara 15y ago
I recently had to scrape the yellow pages for a list of doctors around chicago. My first script caused them to block my ip address. The warning page was cute, it said I had violated one or more of Asimov's 3 laws of robotics.
I wrote a second script that included delays between loading pages (just 10-15 seconds, some longer pauses every 10 pages or so) and it ran for a whole day without being blocked.