4 ms·
Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the
by juanse 5y ago
Nowadays is more and more common for websites to have some kind of rate limiting middleware such as rack attack for ruby. It would be interesting to explore the strategies to deal with it.
- hdjjhhvvhga 5y agoSame as always - proxy farms, random popular UAs with random delays etc.
- juanse 5y agoSorry, what is a UA?
- bryanrasmussen 5y agoso will Google's freezing of the UA lead to less ability to web scrape for the non big company scrapers out there?
- vivekv 5y agoSorry can you please elaborate what is Google doing?
- bryanrasmussen 5y agosorry, I thought it was a well known thing here given the various discussions over past year or so https://groups.google.com/a/chromium.org/g/blink-dev/c/-2JIRNMWJ7s/m/yHe4tQNLCgAJ https://groups.google.com/a/chromium.org/g/blink-dev/c/-2JIR... on edit: so I'm thinking as there will only be one UA floating around then, sure, older UAs can exist, but those become progressively more suspicious.
- hdfinenwvdun 5y agoYou can easily spoof the UA
- dreyfan 5y agoWhat? It’s just a text string in the header. How in the world would that possibly make it more difficult to scrape? All Chrome is doing is stop appending the current semver in the UA it sends.
- bryanrasmussen 5y agoI mean I realize it probably isn't a problematic, just wondering, but on the other hand it shouldn't be so difficult to follow the reasoning based on the context I would think: poster says - in order to be able to scrape effectively you should appear to be a real human, use different UAs etc. So as this change happens different UAs become one less thing that you can easily change to seem less suspicious, as a non-frozen UA would then be a suspicious sign after some time. So a sort of side effect.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- heavyset_go 5y agoThe switch from UA to browser fingerprinting makes it harder to scrape without being stopped. Yes, at any time the UA could be ignored and clients could be fingerprinted, but now the UA is being made next to useless, so fingerprinting will now become the default everywhere.
- Ozzie_osman 5y agoFundamentally, what most scrapers learn is that the more their scraper can behave like a human browsing the site, the less likely they are to get detected and blocked. This does put limits on how quickly they can crawl, of course, but scrapers find ways around it like changing ip and user agent (ip is probably the main one, bec you can then pretend that you are multiple humans browsing the site normally).
- prox 5y agoYeah, there are services that give you a range of IPs for a certain time.
- Nextgrid 5y agoEven changing IPs won't always work against an adversary with a global view of the Internet such as CloudFlare. CF has a view on a significant chunk of internet traffic across many sites and feeds that into some kind of heuristics/machine learning. Even if we assume that your behavior on the scraped website looks human-like, you may still get blocked or challenged because of your lack of traffic on other sites. The IPs you'd get from a typical proxy service would only be used for bot activity and would've been classified as such a long time ago, and there's no "human activity" on it to compensate and muddy the waters so to speak. The best solution is to use IPs with a chunk of legitimate residential traffic, and keep scraping sessions constrained to their IPs - don't rotate your requests among all these IPs, instead every IP should be its own instance of a human-like scraper, using its own user account, browser cookies, etc.
- marvram 5y agoYou nailed it! I've also faced issues in the past with captchas, and elaborate bot detection mechanisms. It would also be helpful to mention that there are automatic captcha solvers to bypass security once one is detected. I am wondering if it is worthwhile to provide an addition to this post on how to improve the efficacy of scraping despite these roadblocks. The article is geared towards beginner scrapers that are just starting out so maybe it would be overkill? What do you think?
- marginalia_nu 5y agoWhy not just lower the crawl rate? My search engine crawler visits tens of million documents in a week at a rate of 1 doc/second, but a few hundred different domains at the same time. Going as low as 0.2 dps could easily be doable I think.
- stef25 5y agoRotate through a bunch of proxies.
- heavyset_go 5y ago> It would be interesting to explore the strategies to deal with it. The strategy to deal with it is to behave well when making requests so that you don't get rate limited.