4 ms·

A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set

by m4r71n 3y ago

A note of caution: never scrape the web from your local/residential network. Few months back I wanted to verify a set of around 200k URLs from a large data set that included a set of URL references for each object, and naively wrote a simple Python script that would use ten concurrent threads to ping each URL and record the HTTP status code that came back. I let this run for some time and was happy with the results, only to find out later that a large CDN provider has identified me as a spammy client with their client reputation score and blocked my IP address on all of the websites that they serve. Changing your IP address with AT&T is a pain (even though the claim is that your IP is dynamic, in practice it hardly ever changes) so I opted to contact the CDN vendor by filling out a form and luckily my ban was lifted in a day or two. Nevertheless, it was annoying that suddenly a quarter of the websites I normally visit were not accessible to me since the CDN covers a large swath of the Internet.

ricardo81 3y ago
It makes very little difference what IP you scrape from, unless you're from a very dodgy subnet. The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.
- remram 3y ago
  I think the emphasis is on "never scrape from YOUR local/residential network".
marginalia_nu 3y ago
I run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot. Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.
delfinom 3y ago
>(even though the claim is that your IP is dynamic, in practice it hardly ever changes) Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it. Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.
- cand0r 3y ago
  When the lease expires, the same IP is prioritized for renewal. Leases are generally for a week or two, but I've noticed dynamic IPs staying for 3 months or more. Swapping modems is really the best way to get a new external IP.
hinkley 3y ago
I’ve learned a bunch of stuff about batch processing in the last few years that I would have sworn I already knew. We had a periodic script that had all of these caveats about checking telemetry on the affected systems before running it, and even when it was happy it took gobs of hardware and ran for over 30 minutes. There were all sorts of mistakes about traffic shaping that made it very bursty, like batching versus rate limiting, so the settings were determined by trial and error, essentially based on the 95th percentile of worst case (which is to say occasionally you’d get unlucky and knock things over). It also had to gather data from three services to feed the fourth and it was very spammy about that as well. I reworked the whole thing with actual rate limiting, some different async blocks to interleave traffic to different services, and some composite rate limiting so we would call service C no faster than Service D could retire requests. At one point I cut the cluster core count by 70% and the run time down to 8 minutes. Around a 12x speed up. Doing exactly the same amount of work, but doing it smarter. CDNs and SaaS companies are in a weird spot where typical spider etiquette falls down. Good spiders limit themselves to N simultaneous requests per domain, trying to balance their burden across the entire internet. But they are capable of M*N total simultaneous requests, and so if you have a narrow domain or get unlucky they can spider twenty of your sites at the same time. Depending on how your cluster works (ie, cache expiry) that may actually cause more stress on the cluster than just blowing up one Host at a time. People can get quite grumpy about this behind closed doors, and punishing the miscreants definitely gets discussed.
pknerd 3y ago
Most probably cloud based scraping services were not available in 2012. Now there are services available like scraperapi and others that don't need you to install anything at your end. You pay them, use their cloud infra, infinite proxies and even headless browsers. Shameless plug, I had written about it a few years ago on my blog post [1] [1] https://blog.adnansiddiqi.me/scraping-dynamic-websites-using-scraper-api-and-python/ https://blog.adnansiddiqi.me/scraping-dynamic-websites-using...
timeon 3y ago
Not sure how is it in python, but what about using something like arti-client? Would it be already blocked?

by m4r71n 3y ago

ricardo81 3y ago

It makes very little difference what IP you scrape from, unless you're from a very dodgy subnet. The major content providers tend to go on a whitelist only based approach, you're either a human-like visitor or facing their anti-scraping methodologies.

remram 3y ago
I think the emphasis is on "never scrape from YOUR local/residential network".

marginalia_nu 3y ago

I run a search engine crawler from my residential network. I get this too sometimes, but a lot of the time the IP shit-listing is temporary. It also seems to happen more often if you don't use a high enough crawl delay, ignore robots.txt, do deep crawls ignoring HTTP 429 errors and so on. You know, overall being a bad bot. Overall, it's not as bad as it seems. I doubt anyone would accidentally damage their IP reputation doing otherwise above-board stuff.

delfinom 3y ago

>(even though the claim is that your IP is dynamic, in practice it hardly ever changes) Every ISP just uses DHCP for router IPs. It's dynamic, you just have to let the lease time expire to renew it. Or, have your own configurable router instead of the ISPs so that you can actually send a dhcp release command though they don't all support this part. Changing MAC Address will work otherwise.

cand0r 3y ago
When the lease expires, the same IP is prioritized for renewal. Leases are generally for a week or two, but I've noticed dynamic IPs staying for 3 months or more. Swapping modems is really the best way to get a new external IP.

hinkley 3y ago

I’ve learned a bunch of stuff about batch processing in the last few years that I would have sworn I already knew. We had a periodic script that had all of these caveats about checking telemetry on the affected systems before running it, and even when it was happy it took gobs of hardware and ran for over 30 minutes. There were all sorts of mistakes about traffic shaping that made it very bursty, like batching versus rate limiting, so the settings were determined by trial and error, essentially based on the 95th percentile of worst case (which is to say occasionally you’d get unlucky and knock things over). It also had to gather data from three services to feed the fourth and it was very spammy about that as well. I reworked the whole thing with actual rate limiting, some different async blocks to interleave traffic to different services, and some composite rate limiting so we would call service C no faster than Service D could retire requests. At one point I cut the cluster core count by 70% and the run time down to 8 minutes. Around a 12x speed up. Doing exactly the same amount of work, but doing it smarter. CDNs and SaaS companies are in a weird spot where typical spider etiquette falls down. Good spiders limit themselves to N simultaneous requests per domain, trying to balance their burden across the entire internet. But they are capable of M*N total simultaneous requests, and so if you have a narrow domain or get unlucky they can spider twenty of your sites at the same time. Depending on how your cluster works (ie, cache expiry) that may actually cause more stress on the cluster than just blowing up one Host at a time. People can get quite grumpy about this behind closed doors, and punishing the miscreants definitely gets discussed.

pknerd 3y ago

Most probably cloud based scraping services were not available in 2012. Now there are services available like scraperapi and others that don't need you to install anything at your end. You pay them, use their cloud infra, infinite proxies and even headless browsers. Shameless plug, I had written about it a few years ago on my blog post [1] [1] https://blog.adnansiddiqi.me/scraping-dynamic-websites-using-scraper-api-and-python/ https://blog.adnansiddiqi.me/scraping-dynamic-websites-using...

timeon 3y ago

Not sure how is it in python, but what about using something like arti-client? Would it be already blocked?