8 ms·
How to scrape anything on the web and not get caught
- sram1337 9y agoI don't know if scrapy handles this, but I've run into issues with sites fingerprinting my browser. Proxies help, but there are other ways to identify site visitors aside from IP addresses.
- JeanMarcS 9y agoRotating user agents ?
- dredmorbius 9y agoMinimally effective, or outright detrimental, in my own limited testing.
- jaytaylor 9y agoSome proxies will also include your IP address in the form of an "X-Forwarded-For" header.
- dingaling 9y agoCan't you use a service like Panopticlick to 'tune' the browser fingerprint to a majority percentile? Particularly if running in clean VM images.
- ikeboy 9y ago>after introducing proxies my crawl times grew by an order of magnitude from minutes to hours Yeah, same experience. Right now I use luminati.io datacenter IPs that work ok, anyone know of a cheaper option that works well? Scraping tens of millions of pages a month.
- dewey 9y agoYou could use https://oxylabs.io/ https://oxylabs.io/ or buy some regular VPN accounts and build a http proxy wrapper around that. Extremely cheap and works well. There’s a lot of existing projects on Github for that too.
- ikeboy 9y agooxylabs.io seems more expensive than luminati, minimum of $178/month. Not clear if they charge for bandwidth. Problem with VPN is it's shared and hard to get a lot of IPs, any specific ones I could get say 100 dedicated US IPs for a reasonable price?
- dewey 9y agoDepends on your scale and what you negotiate. The shared nature of VPN is usually not a huge problem but depends on your use case. Most VPN providers have a better deal for bigger customers so you can just buy multiple accounts in bulk with each having for example 5 connections and use that. For US specifically it often happens that they have 100s of servers and vpn configs so you can build something out of that.
- forcer 9y agoDepending on what you scrape we can help. we have proxies in 170+ countries, please get in touch at www.speedchecker.xyz , the pricing is cheaper than what you say above
- tomarr 9y agoI suppose the economics of it comes in to play in a similar vein to mailchimp, the lower the pricing the more scammier clients and the more IPs they lose to blacklists.
- ikeboy 9y agoNot really. Mail is default bad, you need to build up trust just to get a tiny amount of deliverability. Fetching webpages is default good until you're detected as bad. The thing is, a lot of scraping goes unnoticed. Maybe you get an extra thousand hits here and there. But every spam campaign gets noticed and results in some percentage of spam complaints from users.
- buildbuildbuild 9y agoCareful. Using open proxies could be considered unauthorized access in some jurisdictions. Some of these proxies were installed without the user's permission. This is my favorite consensual alternative: https://github.com/mattes/rotating-proxy https://github.com/mattes/rotating-proxy
- darpa_escapee 9y agoThis solution will not work with HTTPS. There are other alternatives and it's easy to roll your own.
- endisukaj 9y agoFor example?
- Theodores 9y agoI have no idea what tools are available for denying access to web scrapers, this I should know given that I have built a few websites and know what to do to get pages serving quickly. Somehow I missed the memo on how to set your site up to not be scraped. Is there an nginx setting for that? This could be interesting for people that do scrape sites to know too, what basic reasonable measures can one take beyond looking at logs and doing IP bans?
- pjc50 9y agoThe cheap technique is rate limiting, but there are also "canary pages" (linked invisibly but should not be accessed by humans) and other techniques for looking to see if the flow through the site is as expected.
- jstarfish 9y agoIPs are naive and too broad. Force users to authenticate to see your content. Unauthenticated viewers get to see only a scrambled version. Now you can track usage by username. Establish a baseline as to what constitutes "normal" browsing. Any user exceeding a threshold of page requests gets rate limited/banned. Stopping people from recording public information is unrealistic, but you can certainly make it more of a pain in the ass.
- amelius 9y agoCaptchas are one (drastic) option.
- nojvek 9y agoAny image based captcha where one needs to identify words from an image can be easily broken by algorithms now. Google would be the leader with re-captcha, but as a human, I fail a whole number of them. They are a very annoying experience to your users.
- mmcwilliams 9y agoI'm sort of on the other side of this problem, but I think an interesting counter-measure came up in a recent Show HN[0]. IP bans alone are probably not enough to stop a motivated scraper, or work longterm, so checking whether or not traffic is coming from a cloud provider and throwing up a captcha would be a significant hurdle, while allowing humans on cloud-hosted VPNs to pass through without much trouble. [0] https://news.ycombinator.com/item?id=16868012 https://news.ycombinator.com/item?id=16868012
- amelius 9y agoCould this be a solution?: run a website, and let your visitors do the crawling.
- wiradikusuma 9y agoI use paid service proxy bonanza ($12/mo for 2 IPs), and build my own as well using squid ($5/mo DigitalOcean).
- lemagedurage 9y agoYou pay more per IP than per server!? Why don't you get 3 DO instances then?
- wiradikusuma 9y agoThe paid service has IPs from a lot of other countries not covered by DO :(
- chrismeller 9y agoI can't speak for that particular service, but in other similar ones I've looked at and used in the past, you're paying for a certain number of IPs _at a given time_. So, for instance, they have a pool of servers that have 1000 IPs available. Your account allows connections to go out over 2 of those at a time. If something happens (like one gets banned by whatever service you're scraping), you can get a different set of 2 IPs and keep moving. While you're still paying a relatively high price for what you're consuming (predominantly bandwidth in this case), you're paying for the flexibility.
- qiqitori 9y agoI had to scrape past an access limit per IP once, and just went with IPv6 addresses, of which IPv6 users have plenty. (This was almost five years ago, so it's conceivable that some services would have wisened up a bit and block the entire IPv6 block now.)
- pdkl95 9y agoI assume simple BFS and DFS traversal behavior shows up brightly in access logs, making detection more likely. Does it help to use Random First Search[1]? Or is it better to attempt emulating human actors (requires much more develop effort)? [1] https://bl.ocks.org/mbostock/11161648 https://bl.ocks.org/mbostock/11161648
- your-nanny 9y agoWhat sorts of real-world and legitimate/ethical use cases are there for wholesale repeated scraping?
- macsj200 9y agoInternet archival initiatives are a legitimate use case, as well as things like sneakernet in Cuba.
- mickeyp 9y agoScraping websites like AliExpress and Ebay for trademark and brand infringement. There are many vendors who sell fake tat on the internet, and there are companies out there who track it down on behalf of the brand owners. These companies tend to have armies of lawyers who can swat away even the likes of eBay when it comes to justifying web scraping. Nevertheless, the work required is still the same tedium that others deal with: CAPTCHAs; throttling; IP bans; etc.
- salawat 9y agoI'm kind of interested in an answer to this as well. I know the typical "travel site" or "comparison shop" use case. There is also the "darn it, I want this" use case. However, automated, periodic web scraping that mutates (I.e. ticket or reservation grabbing bots) has always felt a bit squicky to me in the same way DNS squatting does.
- megous 9y agoWebsite you like that invites discussion has a very shitty comment system, that doesn't notify you of the responses to your comments. Or perhaps you want to monitor mentions of your name, and join the discussion. Or you want to not lose your past thoughts, because discussions were deep and some may be interesting to re-read in the future. Or a website is known to allow users delete their comments, or website itself bans users and hides all their content from others for flimsy reasons like "ISIS" in the title, no matter whether it's pro/against/neutral/irrelevant. Or you want to organize the information differently than the site does. Or the website is bloated and slow as hell, and you want to use it over gprs, so you create a lightweight/fast/better organized/filtered mirror. Or the website doesn't have search or it is crap/slooow, and is not indexed in google, like some IRC logs out there. Or you want to have content available offline. ...
- CapacitorSet 9y agoThat's a very low-quality article, in my opinion. It takes an entire article to show how to use a simple tool and how to fetch a list of proxies, uses Makefile when a shell script would do just fine, and exaggerates the title.
- time0ut 9y agoI came here to make this same comment. This article barely begins to scratch the surface on this topic. Effective, high volume scraping involves more than just rotating through a list of proxies.
- nkozyra 9y agoIn addition, it does nothing to mitigate even remotely significant scrape detection. Doesn't talk about request fingerprints, patterns, user agents and headers, multi region access, etc. This is a hobbyists' guide to scraping under the radar. Fine at that scale but quite incomplete for anything remotely mature or wide reaching.
- ospider 9y agoI do web crawling for a living, the method mentioned in the article does not work for most sites
- alehul 9y agoI've been working on a project that requires scraping from a large number of sites. Do you have any recommendations on better resources?
- Bromskloss 9y agoDo most sites have scraping detection at all? Are they even opposed to scraping?
- codefined 9y agoOn a website I'd written we had pseudo-randomly generated URLs to show dynamic content (it was a game, the URL contained parameters). On each page we had this little widget that included five random configurations people might like to try. A few times our website went down due to the load going >30, eventually I discovered Google was doing something funky, adding the dynamic domains to the "robot.txt" file fixed the issue. Then some other search engines / scrapers seemed to run into the same issue and started requesting hundreds of thousands of URLs per day (these pages were dynamically generated and took a moderate amount of compute power). We eventually did have to implement basic anti-scraper rules because it was degrading the user experience.
- dis-sys 9y agoThis article has the lowest quality ever seen on HN. The title is hugely exaggerated. Using a list of proxies and hope that is enough to scrape _anything_ on web?
- UniZero 9y agoContrary to popular belief, a lot of high traffic sites can be scrapped by a single IP without hitting access limits.
- mattcoles 9y agoIn what way is the article not scummy as hell? You shouldn't waste Jenkins server time with this...
- robsun 9y agoSome time ago I was looking for an apartment to buy. Sites in my country are bloated and terribly slow. Checking several offers took minutes. Moreover I live in a city where good offers are sold the same day they are published. I decided to run scrapper to fetch all the data about available apartments in my city. Thanks to that I was able to browse offer at speed of tinder. It took me a few hours to write all the stuff, it saved me probably weeks. To avoid getting caught I decided to setup TOR on my raspberry pi and use it as a proxy. It was extremely easy and reliable. Sites were so slow I didn't notice significant performance drop. I didn't care about changing proxies because TOR made it for me. Except that it is good idea to change User-Agents and add some random delays between calls. Luckily for this case it was enough.
- Bromskloss 9y ago> proxy-lists getProxies --sources-white-list="gatherproxy,sockslist" Is there a reason for using only gatherproxy and sockslist? There are more lists [0] available. [0] https://github.com/chill117/proxy-lists/blob/4bb8064703b09ee07488f04eaf992431fdbe7761/readme.md#supported-proxy-lists https://github.com/chill117/proxy-lists/blob/4bb8064703b09ee...