3 ms·

Providers like Oxylabs can be quite restrictive, preventing access to many of the common sites that scrapers choose to target. https://faq.oxylabs.info/en/arti

Providers like Oxylabs can be quite restrictive, preventing access to many of the common sites that scrapers choose to target. https://faq.oxylabs.info/en/articles/8826164-restricted-targets-proxy-solutions-and-web-scraper-api https://faq.oxylabs.info/en/articles/8826164-restricted-targ... Additionally I believe that $4/GB is an introductory price. When I went into the Oxylab dashboard, it showed me $8.

faizshah 2y ago
Hmm that’s unfortunate. I’m actually scaling up a data journalism project this month which is why I’ve been looking at these. I’m curious if you can suggest a happy medium between curl-impersonate on VPS (dirt cheap) vs residential proxy ($8/gb)? Personally I’m not trying to hit any of those common sites.
gruez 2y ago
>Providers like Oxylabs can be quite restrictive, preventing access to many of the common sites that scrapers choose to target. Most of them seem pretty reasonable? "Entertainment & streaming" - who's trying to scrape netflix's library? "Banking and other financial institutions" / "Government websites" / "Mailing" - seems far more likely it'll be used for credential stuffing than for "scraping". "Ticketing" - seems far more likely that it'll get used by scalpers than for scraping The main targets of scraping - e-commerce sites (for price comparisons) and social media networks (for user generated content) are fine to scrape. Is there some use case I'm missing here? Is there a huge contingent of people wanting to scrape ticketmaster or bank of america?
- bdcravens 2y ago
  I used the term "scrapers" pretty loosely, but yes, in many cases they are more bad actors than actual scrapers. However as they say the list may include other sites, I suspect Oxylab adds sites to the list at the site owners' requests (Amazon, Target, etc are likely to be on those lists)

faizshah 2y ago

Hmm that’s unfortunate. I’m actually scaling up a data journalism project this month which is why I’ve been looking at these. I’m curious if you can suggest a happy medium between curl-impersonate on VPS (dirt cheap) vs residential proxy ($8/gb)? Personally I’m not trying to hit any of those common sites.

gruez 2y ago

>Providers like Oxylabs can be quite restrictive, preventing access to many of the common sites that scrapers choose to target. Most of them seem pretty reasonable? "Entertainment & streaming" - who's trying to scrape netflix's library? "Banking and other financial institutions" / "Government websites" / "Mailing" - seems far more likely it'll be used for credential stuffing than for "scraping". "Ticketing" - seems far more likely that it'll get used by scalpers than for scraping The main targets of scraping - e-commerce sites (for price comparisons) and social media networks (for user generated content) are fine to scrape. Is there some use case I'm missing here? Is there a huge contingent of people wanting to scrape ticketmaster or bank of america?

bdcravens 2y ago
I used the term "scrapers" pretty loosely, but yes, in many cases they are more bad actors than actual scrapers. However as they say the list may include other sites, I suspect Oxylab adds sites to the list at the site owners' requests (Amazon, Target, etc are likely to be on those lists)