11 ms·
Ask HN: What do people use to prevent crawlers?
What people use to prevent robots and crawlers from websites?
- ge0 9y agoCan you give a little more context to the question? There are various ways but they would be dependent on the reasons in the first place.
- peterkelly 9y agohttps://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard
- AmrMostafa 9y agorobots.txt is a "sign" that you leave at the door for well-behaving robots, but it doesn't actually make any practical difference when a robot isn't implemented to honor it
- twobyfour 9y agoFor well-bahaved bots, robots.txt. For ill-behaved ones, it depends on why you're trying to block them. Rate throttling, IP blocking, requiring login, or just gating all access to the site with HTTP Basic Auth can all work.
- okket 9y agoA password prompt/captcha? If you do not want to get crawled, do not make it public?
- ThePhysicist 9y agoI assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service): https://blog.cloudflare.com/introducing-scrapeshield-discover-defend-dete/ https://blog.cloudflare.com/introducing-scrapeshield-discove... There's a nice Github repo with some advice on blocking scrapers: https://github.com/JonasCz/How-To-Prevent-Scraping https://github.com/JonasCz/How-To-Prevent-Scraping Finally, you could use a plugin in your Webserver to display a CAPTCHA to visitors from IP addresses that cause a lot of requests to your site. There are many more strategies available (up to creating fake websites / content to lead crawlers astray), but the CAPTCHA solution is the most robust one. It will not be able to protect you against crawlers that use a large source IP pool to access your site though.
- owebmaster 9y agoThey used to use flash, for the good or the bad.
- Kenji 9y agorobots.txt then Zip bombs.
- d33 9y agoWhy was this downvoted?
- bad_user 9y agoThe other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.
- amelius 9y agoBut your crawler must be very slow then.
- pmx 9y agoThey generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.
- inglor 9y agoSure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).
- deleted 9y ago[deleted]
- thinbeige 9y agoYou can't prevent good crawlers. Captchas might help and what Amazon does: Erratic, unpredictable changes of the HTML structure.
- hollander 9y agoIf it's about content, SVG and convert all text to curves. /s
- alexcnwy 9y agoCan just screencap the rendered page and use OCR. If someone wants to scrape your website badly enough, they'll find a way.
- pbhjpbhj 9y agoUse a really bad font with terrible kerning and very similar 1lI and oO0, etc.. /s
- majewsky 9y agoSpellcheckers and error-correcting OCR are a thing. In fact, spellchckers are so well-understood that they're a frequent assignment for CS undergraduates.
- laughfactory 9y agoGreat idea. I'll keep this in my back pocket for future use. The wiley and expensive way is to alter your page
- z92 9y agoI run a cronjob every 5 minutes that parses httpd access log. If there's an IP with abnormally large request number, it blocks it. Most crawlers will make hundreds of requests in five minutes, while legitimate viewers will make be bellow 100.
- inglor 9y agoIf an attacker uses something like Illuminati (or just abuses Tor or another botnet) then they can (and should) distribute HTTP requests across different IPs. This is pretty cheap to do and I've seen it done before in several places I've worked (on the defending side).
- lazyjones 9y agoThis is ineffective and dangerous where ISPs allow switching dynamic IP addresses with no delays. Large german ISPs do this, so your abusive scraper will just continue with a new IP address while some legitimate user, who is unlucky enough to get the abuser's old IP address, is blocked.
- inglor 9y agoWe've used Incapsula (cheap and works, but awful support and service) and Distil (expensive and works, great support but steep pricing). Both worked, both worked well with http downloads and selenium (and common techniques). Neither worked against someone dedicated enough - but there are the usual tricks for bypassing them (which we used, to test our own stuff). We also developed something in-house, but that never helps.
- wrath 9y agoI've built crawlers that retrieve billions of web pages every month. We had a whole team working modifying the crawlers to resolve website changes, to reverse engineer ajax requests and solve complex problems like captcha solvers. Bottom line, if someone wants to crawl your website they will. What you can do, however, is make it hard so that the vast majority of developers can't do it (e.g. My tech crawl billions of pages, but there was a whole team dedicated to keeping it going). If you have money to spend, there's Distill Networks or Incapsula that have good solutions. They block PhantomJS and browsers that use Selenium to navigate websites, as well as rate limit the bots. What I found really affective that some websites do is tarpit bots. That is, slowly increase the number of seconds it takes to return the http request. So after a certain amount of request to your site it takes 30+ seconds for the bot to get the HTML back. The downside is that your web servers need to accept many more incoming connections but the benefit is you'll throttle the bots to an acceptable level. I currently run a website that gets crawled a lot, deadheat.ca. I've written a simple algorithm that tarpits bots. I also throw a captcha every now and then when I see an IP address hits too often over a span of a few minutes. The website is not super popular and, in my case, it's pretty simple to differentiate between a human or bot. Hope this helps...
- mkl 9y ago> I've built crawlers that retrieve billions of web pages every month. Wow, what were you doing with the data?
- wrath 9y agoCompetitive intelligence. Crawling thousands of website, mashing up the data to analyze competitiveness between them, and selling it back. For example, cost of flights. Different websites provide different prices for the same flight. The technology crawls all the prices, combines the data, then resells it back to the websites. Everyone knows everyones prices, keeps competition high, lower prices for consumers.
- ketralnis 9y agoTravel companies pay their GDS for every search they do. It costs so much that it's the primary cost centre for some of them. You were costing them thousands of dollars a day.
- huffmsa 9y agoIf you're getting a lot of crawler traffic, your site probably has information a lot of people find useful, so you should consider finding a way to monetize it. Otherwise, your best bet (hardest to get around in my experience) is monitoring for actual user I/O. Like if someone starts typing in an input field, real humans have to click on it beforehand, and most bots won't. Or if a user clicks next-page without the selector being visible or without scrolling the page at all. Not natural behavior. Think like a human.
- FroshKiller 9y agoYou will create accessibility issues for users if you do this. The bias you'd encode in this idea of "human" behavior doesn't consider assistive software at all. I don't click text inputs when my form-filling plugin enters my personal information on a payment screen. And even if I did, you wouldn't know it if I had JavaScript disabled.
- laughfactory 9y agoAs someone who who does a fair amount of scraping for my job, it's always vastly preferred to simply pay for the data. It's generally more efficient and cost effective then scraping. Scraping is always the last resort: we need the data, it's not available for sale, and it's publicly accessible.
- bmetz 9y agoMy favorite thing was to identify bots and instead of blocking them, switch to a slightly scrambled data set to make the scrape useless but look good to the developer who stole it. It was a ton of fun as a side project. I'd also suggest you add some innocent fake data to your real site and then set up google alerts of all of the above to catch traffic. About 50% of sites would respond positively to an email when you showed them they were hosting fake data. About 90% would take my data down if that was followed up with a stronger C&D. One key is to catch them fast, while they're still a little nervous about showing off their stolen data online.
- Moru 9y agoThis is what we used to do. Then send a large zipfile with schreenshots and other data to the lawyers to handle the contact. Shortly after the scraping usually stopped. The contact and sell access wasnt an option because it was competitors taking the data.
- grogenaut 9y agoI did some scraping for a lawyer back in like 01 from other lawyers. He got a c&d and told me to turn it off (we were done anyway). Funny part was the lawyer on the other side wanted us to return all of the content on disk. Not show what we had copied but literally return it. My lawyer laughed about it. The other lawyer was smart/savvy enough to be effectively using the internet in 01 but didn't really understand the tech. Other funny part is if he had generalized his site outside of law he would have had a major business these days.
- bdcravens 9y ago> Funny part was the lawyer on the other side wanted us to return all of the content on disk. I actually had a client that asked me to record a screenshot of me deleting their files from my computer. (and this was actually a developer making the request)
- laughfactory 9y agoAnd this is why it's an excellent idea to always scrape behind proxies. Never scrape from your own IP, or one easily traced to you.
- monodeldiablo 9y ago"How do I stop all these dinner guests from eating this lovely pie I set out on the table?" I remember working hard on a project for a year, then releasing the data and visualizations online. I was very proud. It was very cool. Almost immediately, we saw grad students and research assistants across the globe scraping our site. I started brainstorming clever ways to fend off the scrapers with a colleague when my boss interrupted. Him: "WTF are you doing?" Me: "We're trying to figure out how to prevent people from scraping our data." Him: "WTF do you want to do that for?" Me: "Uh... to prevent them from stealing our data." Him: "But we put it on the public Web..." Me: "Yeah, but that data took thousands of compute hours to grind out. They're getting a valuable product for free!" Him: "So then pull it from Web." Me: "But then we won't get any sales from people who see that we published this new and exciting-- Oh. I see what you mean." Him: "Yeah, just get a list of the top 20 IP addresses, figure out who's scraping, and hand it off to our sales guys. Scraping ain't free, and our prices aren't high. This is a sales tool, and it's working. Now get back to building shit to make our customers lives easier, not shittier." Sure enough, most of the scrapers chose to pay rather than babysit web crawlers once we pointed out that our price was lower than their time cost. If your data is valuable enough to scrape, it's valuable enough to sell. The only technological way to prevent someone crawling your website is to not put it on a publicly-facing property in the first place. If you're concerned about DoS or bandwidth charges, throttle all users. Otherwise, any attempts to restrict bots is just pissing into the wind, IMHO. Spend your energies on generating real value. Don't engage in an arms racw you're destined to lose.
- huffmsa 9y agoExactly. If you have data so valuable that people want to take it, it's going to be a lot easier to figure out how to sell it to them (they're probably not professional web scrapers, it's just a means to an end.) Than to waste yours, theirs and everyone else's time going tit-for-tat keeping them away from your data.
- lazyjones 9y ago> Sure enough, most of the scrapers chose to pay rather than babysit web crawlers once we pointed out that our price was lower than their time cost. If your data is valuable enough to scrape, it's valuable enough to sell. Cool story, bro. Some scrapers will buy your data if it's good enough, but most can't be identified (good luck with those Tor/AWS/dynamic IP users) and some will just resell your data at a lower price. So as a general strategy against scraping, this is useless.
- lazyjones 9y ago- permanently block Tor exit nodes and relays (some relays are hidden exit nodes) - permanently block known anonymizer service IP addresses - permanently block known server IP address ranges, such as AWS - temporarily (short intervals, 5-15 mins) block IP addresses with typical scraping access patterns (more than 1-2 hits/sec over 30+ secs) - add captchas All of these will cost you a small fraction of legitimate users and are only worth it if scraping puts a strain on your server or kills your business model...
- timbowhite 9y agoI wrote a plugin for node.js/express that performs basic bot detection and bans bots by IP address until they pay you some Bitcoin: project: https://github.com/timbowhite/botbouncer https://github.com/timbowhite/botbouncer simple demo: http://botbouncer.xyz/ http://botbouncer.xyz/ I ran it for awhile on some medium traffic websites that were being heavily scraped. It blocked thousands of IP addresses, but IIRC only received one Bitcoin payment.
- digitalzombie 9y agoYou can use cloudfare but it's a small roadblock. I can still crawl that. Also you can do frontend rendering, it's a bit larger roadblock but you can use phantomJS or something to crawl that. IIRC there is a php framework that mutate your front end code but I'm not sure if it does it enough to stop a generalized xpath... Also I used to work for company where they employ people full time for crawling. It will even notified the company if crawler stopped working so they can update their crawler...
- viraptor 9y agoWhy do you think frontend rendering is harder? Every time I see it, I'm happy because there's a nice API it relies on - I can grab clean, structured data from it rather than trying to extract it from the text.
- mattm 9y agoOne thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for. Imagine if everyday they changed? It would make things a lot more difficult. There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displayed only to bots. The crawler could get smart about it and only use xpaths like the 6th div on the page so maybe in the daily update you could throw in some random useless empty divs and spans in various locations. It's a lot of work to setup but I think you would make scraping almost impossible.
- tomascot 9y agoIt works. But you have to create empty elements, too. That way you also target code that is using xpath. Also, you have to take into account regex.
- eli 9y agoThere are scraping tools that render a whole page with a headless browser and then look at what is actually there. Definitely possible. Think about it this way: you could render the whole page as a JPEG with no computer readable text, but someone could still ocr it.
- detaro 9y agoSome sites do that against adblockers, with mixed success.
- bdcravens 9y agoDepends on your content. If the content is dependable, but the DOM isn't, you can get pretty far with something like XPath's contains(). Calling .text on an element in many parsers will happily return all the child content. Worst case is you call .text on <body>
- ghola2k5 9y agoThe best answer is probably an API.
- crispytx 9y agoI think websites, in general, tend to get a lot of bot traffic. My website doesn't have anything valuable to scrape, but I still get 100 hits from bot traffic every day.
- danpalmer 9y agoI encourage developers thinking of doing this to check that they aren't required to have their website be scraper-friendly first. The company I work for does a large amount of scraping of partner websites, with whom we have contracts that allow us to do it and that someone in their company signed off, but we still get blocked and throttled by tech teams who think they are helping by blocking bots. If we can't scrape a site we just turn off the partner, and that means lost business for them.
- Slobbinson 9y agoYou can instruct search engines on how they should crawl a website, by using a robots.txt file. http://www.inmotionhosting.com/support/website/restricting-bots/how-to-stop-search-engines-from-crawling-your-website http://www.inmotionhosting.com/support/website/restricting-b...
- jeremyliew 9y agoI don't understand why you'd want to stop crawlers. If you didn't want people to see your content, it probably shouldn't be on the public web.
- calafrax 9y agoThere are a variety of methods that can be deployed: 1) request fingerprinting - browser request headers have arbitrary patterns that depend on user agent. matching user agent strings with a database of the request header fingerprints allows you to filter out anyone who is not using a real browser who hasn't taken the time to correctly spoof the headers. this will filter out low skill low energy scrapers and create higher costs. 2) put javascript in the page that tracks mouse movement and pings back. this forces scrapers to simulate mouse movement in a js execution environment or reverse engineer your ping back script. this is a very high hurdle and once again forces much more computationally intensive scraping and also much more sophisticated engineering effort. 3) do access pattern detection. require valid refer headers. don't allow api access without page access, etc. check that assets from page are loaded. etc. 4) use maxmind database and treat as suspicious any access not from a consumer isp. block access from aws, gcp, azure, and other cloud services offering cheap ip rental.
- Rjevski 9y agoDon't prevent them. The same data you let humans access for free should be accessible via bots. If you only want to give out a "reasonable" amount of data, that humans wouldn't usually exceed but bots would, then define a rate-limit that wouldn't inconvenience humans and then apply it for everyone - bot or not. That way you're discriminating based on the amount of data instead of whether it's a bot or not. It will thwart people simply paying humans to scrape the data (which would happen if you magically found a way to block bots) while not inconveniencing humans who use a bot to make their job easier while scraping a reasonable amount of data.
- alecco 9y agoThey kill your bandwidth. For a client's catalog site we discovered crawlers were more than half of the used bandwidth costs.
- Rjevski 9y agoDon't humans kill your bandwidth too? In fact a properly written bot would use less bandwidth as it doesn't care about CSS or images. My solution would solve the issue in a fair way by having a "reasonable usage" limit applied to everyone, bot or not. This also means it can't be defeated by someone paying humans to do the dirty work to bypass bot restrictions.
- Pica_soO 9y agoAdd keywords which are likely to get the crawling company involved into a lawsuit (like the names of persons who suit google to be removed from search).
- whatnotests 9y agoOne cannot stop a determined thief. One technique that bothers me quite a bit is constant random changes in class names or DOM structure, which can make it more difficult. Not impossible but more difficult.
- danielbeeke 9y agoThe stack (https://github.com/omega8cc/boa/ https://github.com/omega8cc/boa/) I am using, uses CSF https://www.configserver.com/cp/csf.html https://www.configserver.com/cp/csf.html This is for Drupal sites. It has a strong firewall (csf) and it has a lot of crawler detections on the nginx configurations. It checks the load and when on high load it blocks the crawlers.
- laurieg 9y agoDomain specific but if you detect a bot you can start giving it false information. For example, a dictionary site. Someone tries to crawl your site after triggering your "This is a bot" code, serve bad data to every 20 requests. Mispell a word, Mislabel a noun as a verb, give an incorrect definition. If you combine this with throttling then the value of scraping your site is greatly reduced. Also, most people won't come up with a super advanced crawler if they never get a "Permission denied, please stop crawling" message.
- laughfactory 9y agoThis is very true. My scraping efforts have become vastly more sophisticated after running into explicit attempts to block me. Now I've got all kinds of bells and whistles, and validate the data returned.
- wcummings 9y agoScraping is my birthright, you'll never stop me.
- debacle 9y agoFor most crawlers, robots.txt will work. For people actually trying, nothing but an IP block and vigilance will help.