5 ms·

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile

That's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.

time0ut 8y ago
I came here to make this same comment. This article barely begins to scratch the surface on this topic. Effective, high volume scraping involves more than just rotating through a list of proxies.
nkozyra 8y ago
In addition, it does nothing to mitigate even remotely significant scrape detection. Doesn't talk about request fingerprints, patterns, user agents and headers, multi region access, etc. This is a hobbyists' guide to scraping under the radar. Fine at that scale but quite incomplete for anything remotely mature or wide reaching.
ospider 8y ago
I do web crawling for a living, the method mentioned in the article does not work for most sites
- alehul 8y ago
  I've been working on a project that requires scraping from a large number of sites. Do you have any recommendations on better resources?
- Bromskloss 8y ago
  Do most sites have scraping detection at all? Are they even opposed to scraping?
  - codefined 8y ago
    On a website I'd written we had pseudo-randomly generated URLs to show dynamic content (it was a game, the URL contained parameters). On each page we had this little widget that included five random configurations people might like to try. A few times our website went down due to the load going >30, eventually I discovered Google was doing something funky, adding the dynamic domains to the "robot.txt" file fixed the issue. Then some other search engines / scrapers seemed to run into the same issue and started requesting hundreds of thousands of URLs per day (these pages were dynamically generated and took a moderate amount of compute power). We eventually did have to implement basic anti-scraper rules because it was degrading the user experience.
- kapauldo 8y ago
  What does work?
robsun 8y ago
Same thoughts. Sadly we live in world when it is good enough to get recruiters' attention. Some time ago there was a post on HN about some bullshit like "everyone should have a blog". Do you know any professionals with exceptional experience that is being shared on their blogs? For example, if someone is interested in .NET I can recommend this one: https://www.wiktorzychla.com https://www.wiktorzychla.com

time0ut 8y ago

I came here to make this same comment. This article barely begins to scratch the surface on this topic. Effective, high volume scraping involves more than just rotating through a list of proxies.

nkozyra 8y ago

In addition, it does nothing to mitigate even remotely significant scrape detection. Doesn't talk about request fingerprints, patterns, user agents and headers, multi region access, etc. This is a hobbyists' guide to scraping under the radar. Fine at that scale but quite incomplete for anything remotely mature or wide reaching.

ospider 8y ago

I do web crawling for a living, the method mentioned in the article does not work for most sites

alehul 8y ago
I've been working on a project that requires scraping from a large number of sites. Do you have any recommendations on better resources?
Bromskloss 8y ago
Do most sites have scraping detection at all? Are they even opposed to scraping?
- codefined 8y ago
  On a website I'd written we had pseudo-randomly generated URLs to show dynamic content (it was a game, the URL contained parameters). On each page we had this little widget that included five random configurations people might like to try. A few times our website went down due to the load going >30, eventually I discovered Google was doing something funky, adding the dynamic domains to the "robot.txt" file fixed the issue. Then some other search engines / scrapers seemed to run into the same issue and started requesting hundreds of thousands of URLs per day (these pages were dynamically generated and took a moderate amount of compute power). We eventually did have to implement basic anti-scraper rules because it was degrading the user experience.
kapauldo 8y ago
What does work?

robsun 8y ago

Same thoughts. Sadly we live in world when it is good enough to get recruiters' attention. Some time ago there was a post on HN about some bullshit like "everyone should have a blog". Do you know any professionals with exceptional experience that is being shared on their blogs? For example, if someone is interested in .NET I can recommend this one: https://www.wiktorzychla.com https://www.wiktorzychla.com