19 ms·
Web Scraping: Bypassing “403 Forbidden,” captchas, and more
- herbst 10y agoI've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.
- tehlike 10y agoAnticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.
- herbst 10y agowhile I agree, it may also is a option for some unskilled people to get to money they really need. IMO its better than buying cheap clothes, as captcha at least does not kill those people. To clearify: I lived in some cheap third world cities, and i can certanly see that solving captchas could finance a rather nice life.
- homakov 10y agoMost of the time they use OCR, humans are unreliable and rarely used.
- tyingq 10y agoThat doesn't seem to match the sales pitch at antigate.com, deathbycaptcha, etc.
- herbst 10y agono, at least antigate doesnt. When you hit recaptcha with known proxy urls (or generally hit it a few times per hour) the captchas get so bad that no OCR would be able to solve it, even humans struggle
- tehlike 10y agoyup, exactly. i tried tesarract before (nothing too fancy), it didn't have problems solving it, but at some point it became really hard. I think part of it is how you crawl (phantomjs, for example, seem to hit captcha almost every time), but things like ip&proxy usage could make this trigger more often.
- bdcravens 10y agoAntigate can solve things that would give OCR fits, like animated letters and the like.
- driverdan 10y agoIt's not exploitation.
- bla2 10y agoNice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.
- richthegeek 10y agoWhat's wrong with British spelling? It's also the English spelling using in India, Australia, New Zealand etc. By pure numbers, more people may spell it defence than defense. Americocentrism is quite annoying from the other side :)
- toxik 10y agoSort of a tangent, but I doubt more people speak and write RE over CE.
- Nadya 10y agoNothing? They were pointing out that the "unfortunately spelled" file is merely using the British way of spelling. I'm wondering how they were misunderstood, given they even used quotations to show they were quoting the article which is the one that made the claim that it was "unfortunately spelled".
- celticninja 10y agoyou are shooting the messenger.
- bla2 10y agoI'm not saying anything's wrong with it. The "unfortunately named" bit is from the article, and I'm just pointing out that the author's snark is ill-placed.
- martin-adams 10y agoWell, I did not know that defense was the American spelling. I spent ages looking at that trying to find out which one was the British and which was the American spelling. You learn something every day.
- m00dy 10y agoI use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.
- ben_jones 10y agoWell the problem is when someone scrapes ALL the good listings then pre-purchases them for resale at double the cost.
- Raphmedia 10y agoHow is it different than paying 50+ low-wage remote workers to "scrape" the phonebook for you and then using the information acquired for profit?
- greglindahl 10y agoOne difference is that Feist v. Rural Telephone says that the data in a phonebook can't be copyrighted. https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._Rural_Telephone_Service_Co https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
- Raphmedia 10y agoWhat about using those employees to "crawl" the web for you then?
- greglindahl 10y agoI suspect it's roughly the same as a crawer -- same issues of fair use, TOS/CFAA, etc -- but likely there's no expectation that humans will read and follow robots.txt.
- retube 10y agoso can you use greasemonkey to follow links, load new page, parse new page, just like a headless crawler?
- superasn 10y agoThe web scraping tool of my choice still has to be WWW::Mechanize for Perl. P.S. I wrote a WWW::Mechanize::Query ext for it so that it supports css selectors etc if anyone is interested. It's on cpan.
- 3pt14159 10y agoSame in Ruby land. That with inspector gadget and you're golden.
- jacquesm 10y agoNote that in some places this constitutes breaking the law.
- brogrammernot 10y agoThis is the first I've heard of this. Where/what laws prohibit web scraping?
- jacquesm 10y agohttps://en.wikipedia.org/wiki/Web_scraping#Legal_issues https://en.wikipedia.org/wiki/Web_scraping#Legal_issues https://www.wired.com/2010/10/hacking-captcha/ https://www.wired.com/2010/10/hacking-captcha/ https://www.wired.com/2010/03/wiseguys-indicted/ https://www.wired.com/2010/03/wiseguys-indicted/ http://www.nj.com/news/index.ssf/2011/06/wiseguy_ticket_operators_get_p.html http://www.nj.com/news/index.ssf/2011/06/wiseguy_ticket_oper...
- cookiecaper 10y agoIt's almost always illegal in the United States. It's prohibited by a combination of the CFAA, copyright law, and contractual obligations imposed by Terms of Use, which are usually considered applicable if you load more than one page ("browsewrap"). The CFAA makes it a crime to access any computer network without authorization or in excess of granted authorization. The Terms of Use will usually prohibit "any automated or mechanical access" or use similar boilerplate that can be construed as a restriction on automated access. The implied license to make a copy of the page in RAM is no longer applicable and the scraper is thus infringing copyright. Relevant cases are Craigslist v 3Taps, Facebook Inc. v Power Ventures Inc., and several others. This is at the point where it's basically well-established. The exception is Perfect 10 v. Amazon, where judges ruled that since it was Google and they don't want to break Google, it's OK. Copyright law allows such evaluations because each judge must decide whether a use was "fair" or not.
- TeMPOraL 10y agoThat's a very, very sad turn of events, and I have to wonder, how did we get there? I'm increasingly feeling that the law is giving way too much control over content published on the Internet to the publishers.
- Exuma 10y agoGreat article!
- dmn001 10y agoThe first part seems like a very long-winded way to say "don't use the default user agent". The captcha was unusually simple to solve, in most cases the best strategy is to avoid seeing it in the first place.
- nip 10y agoScrapy is indeed excellent. One feature that I really like is Scrapy Shell [1]. It allows to run and debug the scraping code without running the spider, right from the CLI. I use it extensively to test that my selectors (both CSS and XPATH) are returning the proper data on a test URL. [1] https://doc.scrapy.org/en/latest/topics/shell.html https://doc.scrapy.org/en/latest/topics/shell.html
- brilliantcode 10y agoA few things turn me off about Scrapy is that it feels over engineered for what it does. Why do I need an entire framework? I'm taking on technical debt to access data I don't have programmatic access to. CSS/Xpath are very fragile. You most likely will be changing them in the future.
- eli 10y ago> CSS/Xpath are very fragile. You most likely will be changing them in the future. Genuinely curious what the alternative is
- brilliantcode 10y agoI've been doing research on this but it's not clear whether this problem is a pain for enough number of businesses to justify further investments. I often feel like web scraping is a commodity without understanding any of the inherent technological complexities and challenges. Very discouraging field to be in, especially when people claim to have pain but are unwilling to pay very much for it or show appreciation for the effort that goes into it. edit: thanks for the downvotes. perfect illustration of how innovation is punished and unrewarded in this field.
- jlgaddis 10y agoFYI, I only downvoted you after you complained about downvotes.
- Lxr 10y agoI have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?
- onli 10y agoThe Scrapy cloud alone is worth it. And it gives guidance, making the start very easy.
- makmanalp 10y agoPluggable parsers, automatically good error handling and spidering functionality (finding and queueing new links to scrape), great logging, progress stats, exports, pause/resume functionality, and a million other goodies that are seemingly "trivial" but really you don't want to rewrite them every time you write a scraper. edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.
- nostrademons 10y agoHow much benefit does the spidering/progress/pause/resume functionality give if you're not just spidering every link on the site, but have complex logic to determine exactly which links to crawl and in what order? Does Scrapy provide convenient extension hooks to change the crawl algorithm?
- un-devmox 10y agoI haven't had the need to use pause/resume but I do incorporate logic (not necessarily all that complex) in determining which links to crawl. It is very easy to do within each spider especially with how the framework uses generators. It is also easy to extend the pipelines for pre and post processing. As others have said, managing a project with Scrapy is super easy and highly configurable with sane out of the box settings.
- 10y ago
- janci 10y agoI use Java with simple task queue and multiple worker threads (scrapy is only singlethreaded, although uses async I/O). Failed tasks are collected into second queue and restarted when needed. Used Jsoup[1] for parsing, proxychains and HAproxy + tor [2] for distributing across multiple IPs. [1] https://jsoup.org/ https://jsoup.org/ [2] https://github.com/mattes/rotating-proxy https://github.com/mattes/rotating-proxy
- janci 10y agoHardest part was synchronization - to end the main thread only if all tasks are done - when every running task can produce multiple new tasks - with limiting the maximum number of running threads - always running the maximum nubmer of threads if possible semaphores to the rescue
- tokenizerrr 10y agoDoesn't ThreadPoolExecutor take care of all of that if you store the returned Future from the submit method? Then you just have the main thread wait for those.
- foxylion 10y agoI'm curious what others use to scrape modern (javascript based) web applications. The old web (html and links) work fine with tools like Scrapy, but for modern applications which rely on javascript this does no longer work. For my last project I used a chrome plugin which controlled the browsers url locations and clicks. Results where transmitted to a backend server. New jobs (clicks, change urls) where retrieved from the server. This worked fine but required some effort to implement. Is there an open source solution which is as helpful as Scrapy but solves the issues provided by modern javascript websites/applications? With tools like Chrome headless this should now be possible, right?
- brianwawok 10y agoSplash https://github.com/scrapy-plugins/scrapy-splash https://github.com/scrapy-plugins/scrapy-splash Runs a little headless browser.
- foxylion 10y agoInteresting, is there a variant which uses chrome? This would also eliminate most scraping protections.
- brilliantcode 10y agoSplash is not chromium I believe. Therefore it's buggy as hell and doesn't render websites that Chrome can as smoothly and easily.
- deleted 10y ago[deleted]
- janci 10y agoMany times it's actually much easier to scrape an JS-based app. You just find the right API calls and you get nicely formatted data (JSON mostly).
- sergiotapia 10y agoI use Elixir and Hound because it has a nice clean API that's not difficult to mess around with. It's really straightforward. https://github.com/HashNuke/hound https://github.com/HashNuke/hound
- chatmasta 10y agoNote that 99% of time, if a web page is worth scraping, it probably has an accompanying mobile app. It's worth downloading the app and running mitmproxy/burp/charles on the traffic to see if it uses a private API. In my experience, it's much easier to scrape the private mobile API than a public website. This way you get nicely formatted JSON and often bypass rate limits.
- xur17 10y agoDefinitely agree. I'd also recommend using mobile sites for scraping as they tend to be simpler and load faster (assuming they contain all of the data you are interested in).
- jedberg 10y agoHow do you deal with the issue that most mobile apps have a baked in security key for their private API? Or am I being naive to think that most apps have that?
- dsacco 10y agoYou reverse engineer the application, or you run it in a debugger. If the app features certificate pinning to block MITM eavesdropping through your own proxy, you either use one of the XPosed Framework libraries that removes it on the fly in a process hook, or you decompile the app, return-void the GetTrustedClient, GetTrustedServer, AcceptedIssuers, etc. functions. If it features HMAC signing, you decompile the app, find the key, reverse engineer the algorithm that chooses and sorts the paramaters for the HMAC function, and rewrite it outside the app. If the key is generated dynamically you reverse engineer that too, and if it's retrieved from native .so files you're going to have a fun time, but it's still quite doable. All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client.
- jedberg 10y ago> All they can do is pile on layers and layers of abstraction to make it painful. They can't make the private API truly private if it requires something shipped with the client. This is totally true, but the original premise was to do it just with a MITM. I was being generous and assuming most apps do dynamic generation of their keys. I'm probably wrong now that I think about it.
- ouid 10y agotoo bad it's named for ovine prions.
- thefifthsetpin 10y agoBetter solution: pay target-site.com to start building an API for you. Pros: * You'll be working with them rather than against them. * Your solution will be far more robust. * It'll be way cheaper, supposing you account for the ongoing maintenance costs of your fragile scraper. * You're eliminating the possibility that you'll have to deal with legal antagonism * Good anti-scraper defenses are far more sophisticated than what the author dealt with. As a trivial example: he didn't even verify that he was following links that would be visible! Cons: * Possible that target-site.com's owners will tell you to get lost, or they are simply unreachable. * Your competitors will gain access to the API methods you funded, perhaps giving them insight into why that data is valuable. Alternative better solution for small one-off data collection needs: contract a low-income person to just manually download the data you need with a normal web browser. Provide a JS bookmarklet to speed their process if the data set is a bit too big for that.
- pjc50 10y ago> pay target-site.com to start building an API for you. When has that ever worked?
- nibbler_death 10y agoNever.
- graphememes 10y agoNever to my personal attempts
- benjamincburns 10y agoI can't cite specific examples because the ones I know about formed confidential business relationships, but I can say with confidence that this works All. The. Time. That said, if you're some small-time researcher who can't offer a compelling business case to make this happen, then it won't be worth their time and they're likely to show you the door. [Note: my implication here is that it's not because you're small time, but it's because by the nature of your work you're not focusing on business drivers which are meaningful to the company/org you're propositioning]. Edit: Also be warned that if you're building a successful business on scraped personal info, you're begging to be served w/ a class action lawsuit (though take that well-salted, because IANAL and all that jazz).
- mirimir 10y agoSometimes it's also necessary to spread requests over numerous IP addresses.
- piker 10y agoProposition: 99% of scraping use cases are eliminated if the scraper agrees to subsequently abide by the target's terms of service.
- jazoom 10y agoGooglebot doesn't abide by 99% of websites' terms of use.
- simplyluke 10y agoBut that's "different" because they've built a $600bn company off it.
- piker 10y agoI agree with the implication here, but (1) it doesn't actually rebut the proposition and (2) some significant fraction of websites crawled by Googlebot have signed up on Google's webmaster tools, which includes its own ToS that likely governs.
- jordif 10y agoGood article! I been doing scraping for the last 10 years and I've seen a lots of differents things to try to avoid us. Also, I'm in the other side protecting websites to ban scrapers, so funny!
- skinnymuch 10y agoI'm in the same position for the first time (protecting against scraping) and honestly I'm kind of blind right now. Which is weird because of how much scraping I've done (okay not that much). Any tips or tricks or blogs you know of off the top of your head for protecting your site?
- corford 10y agoVirtually everything can be easily defeated. The only outfit I've consistently seen put up a good fight is Distil. They do it by acting a little like Cloudflare. They put their servers in front of your www facing endpoints and use ML to mine their global client traffic to identify bot signals (aided by some aggressive in-browser javascript fingerprinting).
- kbenson 10y agoYeah, Distil is the first outfit I've encountered where they've got the model to make it really hard to reliably bypass. It comes down to "I can spend a significant amount of time trying to bypass this, and I would, but they would likely identify and block me again within a few weeks at most.", and it's not worth it when it's only part of what I need to do to scrap some data, and it's their entire job, and they can afford to hire multiple people. The economics are in their favor, and I make it a point not to fight economics when I recognize them, it's rarely sustainable.
- skinnymuch 10y agoDistil is really interesting.
- skinnymuch 10y ago
- jlgaddis 10y agoHere's an idea (although probably an unpopular one around here): if a site is responding to your scraping attempts with 403s -- a.k.a. "Forbidden" -- stop what you're doing and go away.
- skinnymuch 10y agoThis is a very obvious thing to say. Perhaps it's needed to be said, I don't know -- It's just a very obvious counter.
- jlgaddis 10y agoYeah, it seemed obvious... but judging by all of the comments here on how to "bypass" 403s, it actually wasn't obvious at all.
- skinnymuch 10y agoI meant it is obvious in that everyone knows that. But they'll still want to bypass it. So everyone is completely aware of it.
- fiatjaf 10y agoWhat if the target page is blocking by IP address and if even with 20 different IP addresses you wouldn't be able to fetch all the data you need in a month?
- corford 10y agoProfessional proxy services. Price, IP pool size and quality vary hugely but if you're not trying to scrape an aggressively defended target and don't need to make more than a handful of requests per second, 100K IPs will usually be more than enough to circumvent most rate limits and a pool that size can be rented for under $100/month.
- gbrits 10y agoInterested to know where to get 100k proxies for $100/mo. Can you give some options?
- ge96 10y agoWhat if they use that before:after thing where the content takes say a couple of seconds to appear so when you try to scrape the site it appears that nothing is there. I have only used HTMLSimpleDom scraper with PHP at this point.
- known 10y agoTry lynx
- ivanhoe 10y agoWe all do this, but how legal is it? If people end up in prison for pen testing without permission, how safe is to intentionally alter the user-agent, circumvent captchas, javascript and other protections? Can that be considered as hacking a site and stealing the data?
- eapen 10y agoEnjoyed learning this and playing with it. What would you recommend storing this sort of data in? Not too keen on going with the traditional MySQL.
- ic3cold 10y agoHave you seen the sentry antirobot system I can't remember the name exactly but it's a hosted solution that randomly displays captchas, when it senses suspicious(robots) crawling. It's a nightmare, because after you solve 1 captcha it can display 4 more one after the other. They also ban your IP, so oyu need IP rotators. Any workarounds? ic3cold