13 ms·

Detecting Chrome headless

375 points by avastel 9y ago

westoque 9y ago
Your solutions in detecting Chrome headless is good. But someone who really wants to do web scraping or anything similar will use a real browser like Firefox or Chrome run it through xvfb and control it using webdriver and maybe expose it through an API. I find these to be almost undetectable.. The only way you can mitigate this is to do more interesting mitigation techniques. Liie IP detection, Captchas, etc. edit: when I say real browser, I mean running the full browser process including extensions etc.
- dsacco 9y ago
  Interesting. I've done a lot of scraping and I never did this. Do you have a treatise that explains how it's done? I've actually been experimenting with headless Chrome recently because I like it more than PhantomJS. Practically speaking I don't think IP detection is useful at all these days, and the only Captcha that can't be bypassed is Google's most recent version. The much more successful anti-scraping tactic is to use a sophisticated reverse proxy that analyzes all requests to identify patterns and unusual behavior, indiscriminate from the IP source (because any large scale scraping will come from many IPs anyway).
  - heipei 9y ago
    If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
  - IgorPartola 9y ago
    IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?
    - heipei 9y ago
      There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/
      - ant6n 9y ago
        What about tor.
        heipei 9y ago
        TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
        crtasm 9y ago
        I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
        dsacco 9y ago
        Tor would introduce pretty intolerable latency into a professional data crawling setup.
      - ficklepickle 9y ago
        I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...
  - skinnymuch 9y ago
    I'm thinking OP meant if you want to never be detected, but like you said, you usually never get detected anyway. Bypassing captcha automatically or with captcha bypass services where you pay per captcha completion?
  - pebers 9y ago
    It's not terribly hard using Selenium; you can use one of their Docker images (e.g. selenium/standalone-chrome) and connect to that using Webdriver. Having said that, it has flaws compared to headless Chrome (more moving parts, terrible security, extra memory usage / dependencies for the JVM to run Selenium) so unless that camouflage is proving crucial, I'd avoid it where possible. I've recently migrated a project to headless Chrome and definitely prefer it.
    - taeric 9y ago
      I'm curious on what makes the security worse. The rest, I can see. Though, I have to confess I have been more critical of the whole headless browser binary push than I'd care to acknowledge. I just feel like this is solving a problem that didn't actually exist anymore. Selenium has worked solidly for quite a while. And, you almost certainly should test in multiple browsers/scenarios.
      - pebers 9y ago
        The main security issues I know of so far are: - Neither Selenium nor Webdriver (at least the Python client, but I assume others too) support HTTPS at all. - By default it opens browsers configured to accept any SSL certificate. I can see why that's useful for local testing, but it's a terrible choice to default to. - It logs way too much. Logs every keypress sent to it, including passwords. At least the latter two are fixable, but not trivially so, and better defaults with easy client-side options to opt into insecure mode would have been much better. The former is not fixable without patching both the client and server and in 2017 it's pretty poor to not even have an option for.
        taeric 9y ago
        Odd, I'm pretty sure both of the first two are wrong. We used to have to make sure we had legit certs for selenium to work with https. That or configure the Firefox profile to have already accepted the self signed one. And, really, the first two points are clearly contradictory. I'm guessing I just misunderstand what you mean? Do you mean the communication between the driver? I'm curious why ssl would be important there? Should just be done locally and a standard ssh tunnel can help with any remote encryption you might want.
  - kuschku 9y ago
    > and the only Captcha that can't be bypassed is Google's most recent version That can also be easily bypassed (I’ve done that, without actually planning to do so) if you can make your behaviour seem completely human. So if you have a bot that can browse in a way that seems statistically human you can actually get around that. You can still scrape the same datasets – you just need to keep instances of bots, with all cookies around, and have them browse in certain ways. Classify what category a site might likely belong to, put them into buckets of queues, and have bots pull from queues they’re likely to browse, or search for certain terms they’re likely to search for. In my case, this all happened by accidents – I had IRC bots for a few channels, each kept a perpetual session and would visit every site that was linked (to get the title) and would be able to search Google by scraping. One day I was accessing a bot remotely, and told it to access a page that was NoCaptcha protected (because the site wasn’t working on my home system), yet it passed the captchas perfectly fine. Tried a few more times, always worked. So I tried figuring out why it worked.
    - _e 9y ago
      Were the irc bots able to render javascript or were the bots using a cli lib like python requests? I thought that a client which doesnt run javascript would be a huge red flag.
      - kuschku 9y ago
        They used Firefox with off-screen rendering, as otherwise they wouldn’t be able to get the title of many pages – even many blogspot blogs can’t be read without JS anymore.
- ivanhoe 9y ago
  Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.
  - jordanpg 9y ago
    In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text
    - dsacco 9y ago
      To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
      - ivanhoe 9y ago
        The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
- aleem 9y ago
  The simplest and most effective trick is to have a link that is not visible to humans but only to crawlers. When an IP ends up on that link you know it's a crawler. You can then proceed to block it. A honeypot in other words. Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.
  - threatofrain 9y ago
    That sounds like a great trick, but I'd want a solution against someone getting large swathes of IP's banned, because the bot crawled my site and posted the link somewhere.
  - chongli 9y ago
    Be careful with that. Users with screen readers may fall into your trap.
    - stordoff 9y ago
      Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window? Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
  - downandout 9y ago
    I do something similar to this on PPC campaigns, but not for the purpose of blocking IPs. Bots tend to come in waves from a given site. If more than X% of clients from a given referrer wind up on the fake link (or other techniques show that the clients are bots), my code can automatically suspend the display of ads on the offending site for a period of time. If it continually happens, then the site's owner is the likely culprit, so I have a threshold at which that site is automatically and permanently blacklisted from all of my campaigns. I have found that advanced bot mitigation is the single most significant determining factor of PPC ROI, which is a sad commentary on the current state of the paid advertising ecosystem.
- amelius 9y ago
  Do you also use OCR? Because I guess you could defeat webscrapers by making the HTML structurally totally different from what is visually present in the browser window.
  - kbutler 9y ago
    As has been mentioned elsewhere in the thread, you need to be careful about accessibility and screen reader access.
- baybal2 9y ago
  Yes, it will do nothing against sophisticated botters. It has been a standard tactic to run browsers in purpose built VMs. The last bastion were smarter, more disruptive captchas and rendering performance profiling, but even that got grinded by now. The situation is even more grave with mobile ads as you have zero opportunity to shove captchas, or rely on single unique per-system ID or cookie that was vetted outside of embedded webkit environment.
- feelin_googley 9y ago
  Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
  - feelin_googley 9y ago
    One can also use a microcontroller such as Teensy to script timed keyboard and mouse action. Demonstrations have usually been by security researchers, but these tiny boards can be used wherever one wants to avoid the labor of repetitive navigating, typing and mouse clicking. http://samy.pl/usbdriveby http://samy.pl/usbdriveby
- kccqzy 9y ago
  I've done similar detections. Basically just use keyboard dynamics: by measuring the precise timing of keystrokes (key down, key up) you can tell if it is likely typed by a user manually. I haven't done it but I think mouse movements could be used too. Add onMouseMove event listeners all over the page and see how they are triggered.
  - notgood 9y ago
    Yeah, that's why I use Math.random() * 5 and similar stuff in my scrapers. A lot of people access the web through smartphones so only allowing people with mouse movements seems a bit overkill.
- feelin_googley 9y ago
  Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
- yes_or_gnome 9y ago
  Safari has navigator.webdriver, and others will soon follow. The idea was first introduced as a Fx bug, but hasn't gone any where. https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 There's a webdriver protocol mailing list with a long thread that describes how it should be implemented and how people could avoid it; ex. recompile without the feature. I just can't find it right this minute.
- deleted 9y ago
  [deleted]
- tehlike 9y ago
  There is a sevice for solving captchas. Death by captcha is one
- tehlike 9y ago
  There is a sevice for solving captchas. Death by captcha is one
userbinator 9y ago
Those who want a more "authentic" experience would do better to use a real normal browser, and control it from outside.
netsharc 9y ago
Shouldn't the first block of code have "HeadlessChrome" instead of just "Chrome" as the search term?
- avastel 9y ago
  You're right, I changed the code.
tyingq 9y ago
I wonder how many of these were deliberate, and how many were missed. Google has a vested interest in bot detection. And by releasing headless chrome, they killed off some of the competition. (https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuN...)
- captainmuon 9y ago
  Google also has an interest in making an undetectable bot - for their search engine. Undetectable in the sense that GoogleBot should see the exact same page that humans see. They have been using chromium based code, at least on certain sites, for a while now. I wonder if this person is damaging his rankings with these techniques... Of course, Google announces itself as GoogleBot. It wouldn't surprise me if they did a second stealthy crawl to detect cloaking. (But I think they are honest when they say they don't, and just throw cheap human labor at it instead by having people browse suspect sites.)
  - kuschku 9y ago
    > Of course, Google announces itself as GoogleBot. It wouldn't surprise me if they did a second stealthy crawl to detect cloaking. (But I think they are honest when they say they don't, and just throw cheap human labor at it instead by having people browse suspect sites.) They actually run secondary tests that aren’t GoogleBot. They’re quite easy to detect on very low traffic sites. If you only have a few hundred users, all of which you know personally, and suddenly over the range of a few hours a few users using Chrome visit the page, while it’s not linked or findable in any search engine, just shortly after users using the googlebot UA visited it, and with certain usage patterns – it’s quite obvious. Detecting the Android Bouncer’s VM is equally easy, although that only happened by accident because, due to an automated action, my app crashed in that and submitted a crash report that was unusual, and I managed to extract parameters that’d allow detecting it (similar with other android virus scanners), but I only cared about that to be able to split those "devices" into a separate category in the crash tracker (all my apps are GPL licensed, and don’t do anything evil anyway)
shakna 9y ago
> ... to automate malicious tasks. The most common cases are web scraping... I really don't think scraping should fall onto that list. There isn't even a consensus in the IT world whether or not scraping should be able to be legally restricted.
- userbinator 9y ago
  Google also does it all the time... The author has certainly made his position clear, and I disagree with him too.
  - user5994461 9y ago
    Google is allowed to do it. There are laws to allow for indexing of contents.
    - beagle3 9y ago
      There are? Can you point us to an example?
      - cormacrelf 9y ago
        There aren't. At least, certainly not beyond any particular government-operated websites, and even then. There are contracts and terms of use, though. You can block anyone you want from accessing your site, and tell Google they have to pay for the privilege. Think Twitter a few years back.
  - askmike 9y ago
    You can control Google's scraping by specifying a robots.txt, which most people won't do because they want Google to index them. The article means parties who don't care about what you want.
- throwaway2016a 9y ago
  I came here to say that too. Scraping has a great many legitimate uses. Search engines, scientific research, trying to use publicly available data that doesn't have an API. I've had to scrape government websites quite frequently because they often make public information hard to read by other means. That last one is an interesting one. I think one of the most effective way to deter a scraper might be to just provide an API! Now if you were using the scraped data to republish (copyright infringement) or use it to gain a competitive advantage (re-pricing in eCommerce comes to mind) that is a different story.
fforflo 9y ago
Since when is web scraping a "malicious task"?
- leetbulb 9y ago
  Read the ToS of most websites. :)
  - shakna 9y ago
    It's not as clear cut, and the law doesn't seem to be entirely clear either [0]. Many in the tech world don't agree that it should be [1]. I'd say this would be one of the most controversial things to refer to as 'malicious'. [0] https://arstechnica.com/tech-policy/2017/07/linkedin-its-illegal-to-scrape-our-website-without-permission/ https://arstechnica.com/tech-policy/2017/07/linkedin-its-ill... [1] https://news.ycombinator.com/item?id=14891301 https://news.ycombinator.com/item?id=14891301
  - simlevesque 9y ago
    Most ToS don't forbid scraping for "malicious tasks", they just don't allow it for any task.
    - wolco 9y ago
      At what point does a tos stop being enforcable? Would something like by visiting you grant all copyright on any materials the visor has created to this site? Could you demand 10% of yearly revenue for any business that visits and be legally able to retrieve the funds?
  - Dolores12 9y ago
    There is a difference between reading ToS and accepting it. I am not obligated to anything just by downloading webpage from internet.
  - gsich 9y ago
    I don't need to accept those to use the site. (in most cases)
stevefeinstein 9y ago
So again someone wants to punish all the legitimate people using a web site to get some marginal benefit from detecting the remaining <1%. The inevitable false positives don't affect the "malicious" users. Only the legitimate ones. And how much will this bloat the page load by? Adding more code to an already overly large page isn't helping anyone. Just let the web be the web, and stop trying to control it.
- tyingq 9y ago
  Mentioned this in another comment, but for some websites, the scraping problem has real costs. Airline, hotel, stock prices,etc. For some spaces, scaling and paying bandwidth for unconstrained scraping is costly. And not restricting it hurts the legitimate users because the performance sucks. There are also the scrapers blindly looking for vulnerabilities or other unsavory tactics.
  - always_good 9y ago
    Also, what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. You're just not always in a place to scale to the abuse or build something more complex than some simple heuristic filters.
    - josteink 9y ago
      > what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. Often? Based on what data? I find it much more likely you only often notice aggressive scrapers. That however tells you nothing about the behavior of the average web scraper or web scrapers in general.
      - tyingq 9y ago
        The system encourages it. Ingress data is cheap, and so many scrapers just default to high frequency.
  - 9y ago
skinnymuch 9y ago
How many of these can be faked with some additional code with Chrome headless? Regardless as others are saying, using complete Chrome or Firefox with webdriver solves all these, right? Is there a way to detect the webdriver extension? That's the only difference I think from a normal browser.
- gruez 9y ago
  or you know, run chrome in a real desktop environment. sure it might take more resources, but that's definitely cheaper than tracking down the places in chromium where you need to make the changes.
  - skinnymuch 9y ago
    Is it? I was talking about it being more in a collective fashion and pushed into the code or done in a way where it would be easier to integrate it into Chrome headless. I personally already do a lot of Chrome and Firefox in real desktop environment. I love doing it this way. I know I can mimic a real user. It gives me great comfort. Still, it's only cheaper if you're doing all the tracking down yourself. And I never meant for that to be my point.
- DiThi 9y ago
  > How many of these can be faked with some additional code with Chrome headless? All of them. As soon as you can run some JS code before the page does, every single difference can be monkey-patched. There's no way to distinguish native APIs from fake APIs made by someone that knows all ways of detecting them.
  - skinnymuch 9y ago
    Right yeah, of course.
sorenbs 9y ago
Leaving aside for a moment that many "malicious" use cases are actually fairly common and totally legitimate. Headless Chrome is awesome and such a step up from previous automation tools. The Chromeless project provides a nice abstraction and received 8k start in its first two weeks on Github: https://github.com/graphcool/chromeless https://github.com/graphcool/chromeless
revelation 9y ago
The irony of using JavaScript to detect scraping or bots when the majority of them not used to trick ads don't ever execute any of it because they are a better curl.
- xg15 9y ago
  Well, if you're determined to prevent scraping, it's rather easy to hide content from non-JS bots: simply pull in the content via Ajax or "encrypt" is and perform the decryption via JS. So thinking about how to ward off bots that do go the extra mile makes sense. (From a scrape-protection POV at least)
  - heipei 9y ago
    And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a Webassembly computation that will yield the correct URL to fetch. Or what about a Web Worker? There are endless possibilities, and the only sane way to scrape / index the web in 2017 is a full-fledged browser.
  - codedokode 9y ago
    If you try too hard then you can accidently hide content from search engines too.
codedokode 9y ago
What about mining cryptocurrency on a page load as a solution against scrapers?
- heipei 9y ago
  That's like the people pushing Github PRs which will mine $coin in the CI process. But seriously, you can do that, but scrapers will have short timeouts anyway before they abandon the page or consider it loaded, so there's probably not much to be made in terms of profit.
- almost_usual 9y ago
  There are companies that use PoW functions as a deterrent against scrapers. Similar but not mining Bitcoin exactly.
JoshTriplett 9y ago
This looks like a list of bugs that need fixing; ideally, headless Chrome should be completely indistinguishable from ordinary Chrome, so that it gets an identical view of the web.
- derefr 9y ago
  And, indeed, even if it wasn't, you could always do your malicious scraping et al using regular Chrome with a remote-control extension or OS Accessibility API-based automation. It's kind of pointless to detect headless browsers specifically, if you'll still have the same problems from automated headed browsers. Still, I agree that if people are going to try detecting headless Chrome, Chrome should strive to thwart that. The attacks in the OP seem like low-hanging fruit; I was expecting something more akin to timing attacks on how long Ready events take to fire given delays from actual rendering. Writing the code to imitate that would be a fun week.
- heipei 9y ago
  It depends on the target audience. For Google (and for most people) the goal of Headless Chrome is to offer an easy and feature-complete way of automatically testing websites, e.g. for performance (PWA are all the craze) and bugs. For those folks, it doesn't matter that you can detect the Headless Browser, it only matters that it's working like the regular one 99% of the time. This is a huge step-up from previous technologies like PhantomJS or laborious solutions involving webdriver and many moving components. In some cases they don't even want it to behave exactly like the regular browser. As soon as your website uses any client-side state (cookies, IndexedDB, HTTP caching, service workers, local storage) you want to have to an easy "give me a clean and isolated browsing session" switch like Headless offers. People scraping the web are not the target audience of this.
  - avip 9y ago
    I tend to disagree with your stance on Google's incentives. I'm pretty confident (and have zero factual ground for this) it is exactly taking over the bots industry with a powerful headless browser, that pushed them to release headless chrome. Mind you Google is extremely interested in bots...
XCSme 9y ago
If someone wants to scrape your site he will do it, just find workarounds against your "protection". It is impossible to tell the difference between a real user and an automated scrape request, you can only make their job a bit harder.
- heipei 9y ago
  True. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scraper will need close monitoring to ensure his scripts / regexes are still working and he will likely need a person dedicated to implementing new workarounds as soon as the sites-to-be-scraped push out a new obfuscation method.
  - afpx 9y ago
    In which case, why not just provide a paid API? The content provider will then make extra revenue that would otherwise go to the endless arms race. As others have mentioned, there is nothing (that I know of) that can thwart a motivated and resourceful scraper.
    - heipei 9y ago
      Some services have actually gone down that exact road. pastebin.com is one of them. But in other cases you simply don't want anyone to be able to extract your info automatically. A good example would be e-commerce sites which don't want anyone to be able to scrape their pricing information in large scale and real time.
- almost_usual 9y ago
  Maybe, anti scraping is already a multi million dollar industry with active development from startups and bigger names like Akamai. There are plenty of websites that want scrapers to disappear.
asveikau 9y ago
I am not a web developer type, but making inferences based on the absence of plugins seems morally wrong. The image hack also seems quite imperfect. What if the internet connection craps out in the middle of the page load? You're going to act on that in js and decide the user is bad? Edit: re-read the article and it seems that case would still work. But deciding the user is bad based on some arbitrary metric still seems like something one should have care with. Edit: downvotes confirm web kids don't give a damn that you use flawed heuristics to determine someone is "bad". Some will copy paste this into their project and the unfortunate user who happens to have a less common configuration will be screwed.
- oliv__ 9y ago
  No, downvotes confirm that you are stigmatizing a good chunk of the readership of this website who will not care to read further into your comment than the first 7 words.
  - Spooky23 9y ago
    He lacks tact, but has a point.
    - asveikau 9y ago
      Thanks. Just to clarify, I believe that if someone posts code snippets like this, purporting to do thing X by doing roundabout and overly simplistic thing Y, you really have to own it and assume tons of responsibility for misleading fellow developers who may be unaware of possible edge cases. When somebody copy-pastes a quick solution and it doesn't work for literally everybody, that translates to real people getting a bad deal when they hit your bugs, something we should all as responsible developers seek to avoid. And yeah, I will not waste time being tactful when describing this situation, because I have seen it before and it kind of sucks.
PascLeRasc 9y ago
I don't want to start an argument here, but can someone explain why web scraping is considered malicious?
- sillysaurus3 9y ago
  Many websites have ToS's that forbid scraping.
  - deleted 9y ago
    [deleted]
- zeta0134 9y ago
  I don't believe that responsible web scraping is malicious, but my belief relies on the assumption that the web is meant to be open. That is, information put on the internet that is publicly available is considered free to access, store, and later retransmit. The original web was designed for researchers to share their work, while the modern internet built on top of that platform has other moral systems that don't necessarily agree. Anyway, on a purely technical level, scraping of publicly available content isn't inherently bad unless you're asked to stop, or are scraping so quickly as to cause a service disruption by tying up the target systems. There is nothing malicious about generating normal traffic at the rate of a regular user. The animosity arises from what you plan to do with the data, and whether the entity you're scraping agrees with your usage.
  - PascLeRasc 9y ago
    Thanks, that's what seems obvious to me too that it's just public data and it's possible to collect it without overwhelming the server with requests. I just don't get why someone wouldn't want you to look at their website.
fiatjaf 9y ago
Isn't it possible to detect a bot by tracking some events like random mouse moving, scrolling, clicking etc.? Why weren't these kinds of detection tried in place of captchas, for example?
- kuschku 9y ago
  Because they are easily faked. Google’s current captcha system does track a few of these, but it mostly takes your browsing history, and, if that seems normal, will accept you. I’ve run a few IRC bots that allowed people to submit Google searches, and would return the first resulting link. They also fetch any link mentioned in IRC channels, execute the JS, and after a timeout of 400ms respond with the current page title. Both combined – a normal search history, reading a few hundred pages and videos a day per user – apparently are enough that they seem "human", and can pass NoCaptcha.
tomatsu 9y ago
> var body = document.getElementsByTagName("body")[0]; You can just use document.body. I also suggest to use a data URL instead. E.g. "data:," is an empty plain text file, which, as you can imagine, won't be interpreted as a valid image. let image = new Image(); image.onerror = () => { console.log(image.width); // 0 -> headless }; document.body.appendChild(image); image.src = 'data:,'; > In case of a vanilla Chrome, the image has a width and height that depends on the zoom of the browser The zoom doesn't affect this. It's always in CSS "pixels".
askvictor 9y ago
All of these could quite easily be overcome by compiling your own headless chrome. It wouldn't surprise me if there is a fork to this effect soon.
assafmo 9y ago
"... a headless browser can also be used to automate malicious tasks. The most common cases are web scraping... " Since when web scraping considered malicious? Companies like Google are doing billions because they use web scraping.
DannyDaemonic 9y ago
I'd be willing to bet that missing image size variance is more of a bug or oversight, and is something that will be fixed.
albertgonz14 9y ago
Need a professional hacker that is dedicated to Internet security and all things that deal with digital security ?,..I specialize in most fields of Apple ,Andriod,Microsoft,partners sincerity,employee's honesty,recover your email password,social networks,university grades changing/admin(staff) account hack-access/password,facebook,Instagram,bbm,Skype,snapchat,zoosk,match,eharmony,various blogs,clearing of criminal records,hack gmail accounts,yahoomail,hotmail,aol..gain access to bank accounts several other platforms that constantly run test on different networks and operating system to ensure complete security and functionality .all the services we offer are LEGAL..contact email:Albertgonzalez11960@gmail.com ,whatsapp:+15157170625
hossbeast 9y ago
"Beyond the two harmless use cases given previously, a headless browser can also be used to automate malicious tasks. The most common cases are web scraping, increase advertisement impressions or look for vulnerabilities on a website." Cheating an advertiser I'll grant you, but the other two are 100% legitimate.
josteink 9y ago
> Beyond the two harmless use cases given previously, a headless browser can also be used to automate malicious tasks. The most common cases are web scraping I guess I disagree with the premise of this article. How is web scraping fundamental malicious? What rights/expectations can you have that a publicly accessible website you create must be used by humans only?
- sumedh 9y ago
  It puts a load on you server when bots go wild on your site which in turn affects the experience of legitimate human users of the site.
tscs37 9y ago
I do hope that these methods get patched, I tend to archive my bookmark collection with chrome headless to prevent loosing content when such a site goes offline. I hate it when a website requires me to play special snowflake to scrape them for this purpose.
jdc0589 9y ago
dumb question from someone who's written a ton of scrapers and scraping based "products" for fun: at what point does it make more sense for companies to just start offering open APIs or data exports? Obviously it would never make sense for a company who's value IS their data, but for retail platforms, auction sites, forum platforms, etc... that have a scraper problem, it seems like just providing their useful data through a more controlled, and optimized, avenue could be worth it. The answer is probably "never", it's just something that comes to mind sometimes.
megamindbrian 9y ago
Can you guys shut up already?

375 points by avastel 9y ago

Your solutions in detecting Chrome headless is good. But someone who really wants to do web scraping or anything similar will use a real browser like Firefox or Chrome run it through xvfb and control it using webdriver and maybe expose it through an API. I find these to be almost undetectable.. The only way you can mitigate this is to do more interesting mitigation techniques. Liie IP detection, Captchas, etc. edit: when I say real browser, I mean running the full browser process including extensions etc.

dsacco 9y ago
Interesting. I've done a lot of scraping and I never did this. Do you have a treatise that explains how it's done? I've actually been experimenting with headless Chrome recently because I like it more than PhantomJS. Practically speaking I don't think IP detection is useful at all these days, and the only Captcha that can't be bypassed is Google's most recent version. The much more successful anti-scraping tactic is to use a sophisticated reverse proxy that analyzes all requests to identify patterns and unusual behavior, indiscriminate from the IP source (because any large scale scraping will come from many IPs anyway).
- heipei 9y ago
  If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
- IgorPartola 9y ago
  IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?
  - heipei 9y ago
    There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/
    - ant6n 9y ago
      What about tor.
      - heipei 9y ago
        TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
        crtasm 9y ago
        I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
      - dsacco 9y ago
        Tor would introduce pretty intolerable latency into a professional data crawling setup.
    - ficklepickle 9y ago
      I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...
- skinnymuch 9y ago
  I'm thinking OP meant if you want to never be detected, but like you said, you usually never get detected anyway. Bypassing captcha automatically or with captcha bypass services where you pay per captcha completion?
- pebers 9y ago
  It's not terribly hard using Selenium; you can use one of their Docker images (e.g. selenium/standalone-chrome) and connect to that using Webdriver. Having said that, it has flaws compared to headless Chrome (more moving parts, terrible security, extra memory usage / dependencies for the JVM to run Selenium) so unless that camouflage is proving crucial, I'd avoid it where possible. I've recently migrated a project to headless Chrome and definitely prefer it.
  - taeric 9y ago
    I'm curious on what makes the security worse. The rest, I can see. Though, I have to confess I have been more critical of the whole headless browser binary push than I'd care to acknowledge. I just feel like this is solving a problem that didn't actually exist anymore. Selenium has worked solidly for quite a while. And, you almost certainly should test in multiple browsers/scenarios.
    - pebers 9y ago
      The main security issues I know of so far are: - Neither Selenium nor Webdriver (at least the Python client, but I assume others too) support HTTPS at all. - By default it opens browsers configured to accept any SSL certificate. I can see why that's useful for local testing, but it's a terrible choice to default to. - It logs way too much. Logs every keypress sent to it, including passwords. At least the latter two are fixable, but not trivially so, and better defaults with easy client-side options to opt into insecure mode would have been much better. The former is not fixable without patching both the client and server and in 2017 it's pretty poor to not even have an option for.
      - taeric 9y ago
        Odd, I'm pretty sure both of the first two are wrong. We used to have to make sure we had legit certs for selenium to work with https. That or configure the Firefox profile to have already accepted the self signed one. And, really, the first two points are clearly contradictory. I'm guessing I just misunderstand what you mean? Do you mean the communication between the driver? I'm curious why ssl would be important there? Should just be done locally and a standard ssh tunnel can help with any remote encryption you might want.
- kuschku 9y ago
  > and the only Captcha that can't be bypassed is Google's most recent version That can also be easily bypassed (I’ve done that, without actually planning to do so) if you can make your behaviour seem completely human. So if you have a bot that can browse in a way that seems statistically human you can actually get around that. You can still scrape the same datasets – you just need to keep instances of bots, with all cookies around, and have them browse in certain ways. Classify what category a site might likely belong to, put them into buckets of queues, and have bots pull from queues they’re likely to browse, or search for certain terms they’re likely to search for. In my case, this all happened by accidents – I had IRC bots for a few channels, each kept a perpetual session and would visit every site that was linked (to get the title) and would be able to search Google by scraping. One day I was accessing a bot remotely, and told it to access a page that was NoCaptcha protected (because the site wasn’t working on my home system), yet it passed the captchas perfectly fine. Tried a few more times, always worked. So I tried figuring out why it worked.
  - _e 9y ago
    Were the irc bots able to render javascript or were the bots using a cli lib like python requests? I thought that a client which doesnt run javascript would be a huge red flag.
    - kuschku 9y ago
      They used Firefox with off-screen rendering, as otherwise they wouldn’t be able to get the title of many pages – even many blogspot blogs can’t be read without JS anymore.
ivanhoe 9y ago
Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.
- jordanpg 9y ago
  In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text
  - dsacco 9y ago
    To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
    - ivanhoe 9y ago
      The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
aleem 9y ago
The simplest and most effective trick is to have a link that is not visible to humans but only to crawlers. When an IP ends up on that link you know it's a crawler. You can then proceed to block it. A honeypot in other words. Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.
- threatofrain 9y ago
  That sounds like a great trick, but I'd want a solution against someone getting large swathes of IP's banned, because the bot crawled my site and posted the link somewhere.
- chongli 9y ago
  Be careful with that. Users with screen readers may fall into your trap.
  - stordoff 9y ago
    Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window? Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
- downandout 9y ago
  I do something similar to this on PPC campaigns, but not for the purpose of blocking IPs. Bots tend to come in waves from a given site. If more than X% of clients from a given referrer wind up on the fake link (or other techniques show that the clients are bots), my code can automatically suspend the display of ads on the offending site for a period of time. If it continually happens, then the site's owner is the likely culprit, so I have a threshold at which that site is automatically and permanently blacklisted from all of my campaigns. I have found that advanced bot mitigation is the single most significant determining factor of PPC ROI, which is a sad commentary on the current state of the paid advertising ecosystem.
amelius 9y ago
Do you also use OCR? Because I guess you could defeat webscrapers by making the HTML structurally totally different from what is visually present in the browser window.
- kbutler 9y ago
  As has been mentioned elsewhere in the thread, you need to be careful about accessibility and screen reader access.
baybal2 9y ago
Yes, it will do nothing against sophisticated botters. It has been a standard tactic to run browsers in purpose built VMs. The last bastion were smarter, more disruptive captchas and rendering performance profiling, but even that got grinded by now. The situation is even more grave with mobile ads as you have zero opportunity to shove captchas, or rely on single unique per-system ID or cookie that was vetted outside of embedded webkit environment.
feelin_googley 9y ago
Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
- feelin_googley 9y ago
  One can also use a microcontroller such as Teensy to script timed keyboard and mouse action. Demonstrations have usually been by security researchers, but these tiny boards can be used wherever one wants to avoid the labor of repetitive navigating, typing and mouse clicking. http://samy.pl/usbdriveby http://samy.pl/usbdriveby
kccqzy 9y ago
I've done similar detections. Basically just use keyboard dynamics: by measuring the precise timing of keystrokes (key down, key up) you can tell if it is likely typed by a user manually. I haven't done it but I think mouse movements could be used too. Add onMouseMove event listeners all over the page and see how they are triggered.
- notgood 9y ago
  Yeah, that's why I use Math.random() * 5 and similar stuff in my scrapers. A lot of people access the web through smartphones so only allowing people with mouse movements seems a bit overkill.
feelin_googley 9y ago
Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
yes_or_gnome 9y ago
Safari has navigator.webdriver, and others will soon follow. The idea was first introduced as a Fx bug, but hasn't gone any where. https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 There's a webdriver protocol mailing list with a long thread that describes how it should be implemented and how people could avoid it; ex. recompile without the feature. I just can't find it right this minute.
deleted 9y ago
[deleted]
tehlike 9y ago
There is a sevice for solving captchas. Death by captcha is one
tehlike 9y ago
There is a sevice for solving captchas. Death by captcha is one

userbinator 9y ago

Those who want a more "authentic" experience would do better to use a real normal browser, and control it from outside.

netsharc 9y ago

Shouldn't the first block of code have "HeadlessChrome" instead of just "Chrome" as the search term?

avastel 9y ago
You're right, I changed the code.

tyingq 9y ago

I wonder how many of these were deliberate, and how many were missed. Google has a vested interest in bot detection. And by releasing headless chrome, they killed off some of the competition. (https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuN...)

captainmuon 9y ago
Google also has an interest in making an undetectable bot - for their search engine. Undetectable in the sense that GoogleBot should see the exact same page that humans see. They have been using chromium based code, at least on certain sites, for a while now. I wonder if this person is damaging his rankings with these techniques... Of course, Google announces itself as GoogleBot. It wouldn't surprise me if they did a second stealthy crawl to detect cloaking. (But I think they are honest when they say they don't, and just throw cheap human labor at it instead by having people browse suspect sites.)
- kuschku 9y ago
  > Of course, Google announces itself as GoogleBot. It wouldn't surprise me if they did a second stealthy crawl to detect cloaking. (But I think they are honest when they say they don't, and just throw cheap human labor at it instead by having people browse suspect sites.) They actually run secondary tests that aren’t GoogleBot. They’re quite easy to detect on very low traffic sites. If you only have a few hundred users, all of which you know personally, and suddenly over the range of a few hours a few users using Chrome visit the page, while it’s not linked or findable in any search engine, just shortly after users using the googlebot UA visited it, and with certain usage patterns – it’s quite obvious. Detecting the Android Bouncer’s VM is equally easy, although that only happened by accident because, due to an automated action, my app crashed in that and submitted a crash report that was unusual, and I managed to extract parameters that’d allow detecting it (similar with other android virus scanners), but I only cared about that to be able to split those "devices" into a separate category in the crash tracker (all my apps are GPL licensed, and don’t do anything evil anyway)

shakna 9y ago

> ... to automate malicious tasks. The most common cases are web scraping... I really don't think scraping should fall onto that list. There isn't even a consensus in the IT world whether or not scraping should be able to be legally restricted.

userbinator 9y ago
Google also does it all the time... The author has certainly made his position clear, and I disagree with him too.
- user5994461 9y ago
  Google is allowed to do it. There are laws to allow for indexing of contents.
  - beagle3 9y ago
    There are? Can you point us to an example?
    - cormacrelf 9y ago
      There aren't. At least, certainly not beyond any particular government-operated websites, and even then. There are contracts and terms of use, though. You can block anyone you want from accessing your site, and tell Google they have to pay for the privilege. Think Twitter a few years back.
- askmike 9y ago
  You can control Google's scraping by specifying a robots.txt, which most people won't do because they want Google to index them. The article means parties who don't care about what you want.
throwaway2016a 9y ago
I came here to say that too. Scraping has a great many legitimate uses. Search engines, scientific research, trying to use publicly available data that doesn't have an API. I've had to scrape government websites quite frequently because they often make public information hard to read by other means. That last one is an interesting one. I think one of the most effective way to deter a scraper might be to just provide an API! Now if you were using the scraped data to republish (copyright infringement) or use it to gain a competitive advantage (re-pricing in eCommerce comes to mind) that is a different story.

fforflo 9y ago

Since when is web scraping a "malicious task"?

leetbulb 9y ago
Read the ToS of most websites. :)
- shakna 9y ago
  It's not as clear cut, and the law doesn't seem to be entirely clear either [0]. Many in the tech world don't agree that it should be [1]. I'd say this would be one of the most controversial things to refer to as 'malicious'. [0] https://arstechnica.com/tech-policy/2017/07/linkedin-its-illegal-to-scrape-our-website-without-permission/ https://arstechnica.com/tech-policy/2017/07/linkedin-its-ill... [1] https://news.ycombinator.com/item?id=14891301 https://news.ycombinator.com/item?id=14891301
- simlevesque 9y ago
  Most ToS don't forbid scraping for "malicious tasks", they just don't allow it for any task.
  - wolco 9y ago
    At what point does a tos stop being enforcable? Would something like by visiting you grant all copyright on any materials the visor has created to this site? Could you demand 10% of yearly revenue for any business that visits and be legally able to retrieve the funds?
- Dolores12 9y ago
  There is a difference between reading ToS and accepting it. I am not obligated to anything just by downloading webpage from internet.
- gsich 9y ago
  I don't need to accept those to use the site. (in most cases)

stevefeinstein 9y ago

So again someone wants to punish all the legitimate people using a web site to get some marginal benefit from detecting the remaining <1%. The inevitable false positives don't affect the "malicious" users. Only the legitimate ones. And how much will this bloat the page load by? Adding more code to an already overly large page isn't helping anyone. Just let the web be the web, and stop trying to control it.

tyingq 9y ago
Mentioned this in another comment, but for some websites, the scraping problem has real costs. Airline, hotel, stock prices,etc. For some spaces, scaling and paying bandwidth for unconstrained scraping is costly. And not restricting it hurts the legitimate users because the performance sucks. There are also the scrapers blindly looking for vulnerabilities or other unsavory tactics.
- always_good 9y ago
  Also, what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. You're just not always in a place to scale to the abuse or build something more complex than some simple heuristic filters.
  - josteink 9y ago
    > what I've learned is how little regard for your site your scrapers often have, scraping as aggressively as possible. Often? Based on what data? I find it much more likely you only often notice aggressive scrapers. That however tells you nothing about the behavior of the average web scraper or web scrapers in general.
    - tyingq 9y ago
      The system encourages it. Ingress data is cheap, and so many scrapers just default to high frequency.
- 9y ago

skinnymuch 9y ago

How many of these can be faked with some additional code with Chrome headless? Regardless as others are saying, using complete Chrome or Firefox with webdriver solves all these, right? Is there a way to detect the webdriver extension? That's the only difference I think from a normal browser.

gruez 9y ago
or you know, run chrome in a real desktop environment. sure it might take more resources, but that's definitely cheaper than tracking down the places in chromium where you need to make the changes.
- skinnymuch 9y ago
  Is it? I was talking about it being more in a collective fashion and pushed into the code or done in a way where it would be easier to integrate it into Chrome headless. I personally already do a lot of Chrome and Firefox in real desktop environment. I love doing it this way. I know I can mimic a real user. It gives me great comfort. Still, it's only cheaper if you're doing all the tracking down yourself. And I never meant for that to be my point.
DiThi 9y ago
> How many of these can be faked with some additional code with Chrome headless? All of them. As soon as you can run some JS code before the page does, every single difference can be monkey-patched. There's no way to distinguish native APIs from fake APIs made by someone that knows all ways of detecting them.
- skinnymuch 9y ago
  Right yeah, of course.

sorenbs 9y ago

Leaving aside for a moment that many "malicious" use cases are actually fairly common and totally legitimate. Headless Chrome is awesome and such a step up from previous automation tools. The Chromeless project provides a nice abstraction and received 8k start in its first two weeks on Github: https://github.com/graphcool/chromeless https://github.com/graphcool/chromeless

revelation 9y ago

The irony of using JavaScript to detect scraping or bots when the majority of them not used to trick ads don't ever execute any of it because they are a better curl.

xg15 9y ago
Well, if you're determined to prevent scraping, it's rather easy to hide content from non-JS bots: simply pull in the content via Ajax or "encrypt" is and perform the decryption via JS. So thinking about how to ward off bots that do go the extra mile makes sense. (From a scrape-protection POV at least)
- heipei 9y ago
  And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a Webassembly computation that will yield the correct URL to fetch. Or what about a Web Worker? There are endless possibilities, and the only sane way to scrape / index the web in 2017 is a full-fledged browser.
- codedokode 9y ago
  If you try too hard then you can accidently hide content from search engines too.

codedokode 9y ago

What about mining cryptocurrency on a page load as a solution against scrapers?

heipei 9y ago
That's like the people pushing Github PRs which will mine $coin in the CI process. But seriously, you can do that, but scrapers will have short timeouts anyway before they abandon the page or consider it loaded, so there's probably not much to be made in terms of profit.
almost_usual 9y ago
There are companies that use PoW functions as a deterrent against scrapers. Similar but not mining Bitcoin exactly.

JoshTriplett 9y ago

This looks like a list of bugs that need fixing; ideally, headless Chrome should be completely indistinguishable from ordinary Chrome, so that it gets an identical view of the web.

derefr 9y ago
And, indeed, even if it wasn't, you could always do your malicious scraping et al using regular Chrome with a remote-control extension or OS Accessibility API-based automation. It's kind of pointless to detect headless browsers specifically, if you'll still have the same problems from automated headed browsers. Still, I agree that if people are going to try detecting headless Chrome, Chrome should strive to thwart that. The attacks in the OP seem like low-hanging fruit; I was expecting something more akin to timing attacks on how long Ready events take to fire given delays from actual rendering. Writing the code to imitate that would be a fun week.
heipei 9y ago
It depends on the target audience. For Google (and for most people) the goal of Headless Chrome is to offer an easy and feature-complete way of automatically testing websites, e.g. for performance (PWA are all the craze) and bugs. For those folks, it doesn't matter that you can detect the Headless Browser, it only matters that it's working like the regular one 99% of the time. This is a huge step-up from previous technologies like PhantomJS or laborious solutions involving webdriver and many moving components. In some cases they don't even want it to behave exactly like the regular browser. As soon as your website uses any client-side state (cookies, IndexedDB, HTTP caching, service workers, local storage) you want to have to an easy "give me a clean and isolated browsing session" switch like Headless offers. People scraping the web are not the target audience of this.
- avip 9y ago
  I tend to disagree with your stance on Google's incentives. I'm pretty confident (and have zero factual ground for this) it is exactly taking over the bots industry with a powerful headless browser, that pushed them to release headless chrome. Mind you Google is extremely interested in bots...

XCSme 9y ago

If someone wants to scrape your site he will do it, just find workarounds against your "protection". It is impossible to tell the difference between a real user and an automated scrape request, you can only make their job a bit harder.

heipei 9y ago
True. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scraper will need close monitoring to ensure his scripts / regexes are still working and he will likely need a person dedicated to implementing new workarounds as soon as the sites-to-be-scraped push out a new obfuscation method.
- afpx 9y ago
  In which case, why not just provide a paid API? The content provider will then make extra revenue that would otherwise go to the endless arms race. As others have mentioned, there is nothing (that I know of) that can thwart a motivated and resourceful scraper.
  - heipei 9y ago
    Some services have actually gone down that exact road. pastebin.com is one of them. But in other cases you simply don't want anyone to be able to extract your info automatically. A good example would be e-commerce sites which don't want anyone to be able to scrape their pricing information in large scale and real time.
almost_usual 9y ago
Maybe, anti scraping is already a multi million dollar industry with active development from startups and bigger names like Akamai. There are plenty of websites that want scrapers to disappear.

asveikau 9y ago

I am not a web developer type, but making inferences based on the absence of plugins seems morally wrong. The image hack also seems quite imperfect. What if the internet connection craps out in the middle of the page load? You're going to act on that in js and decide the user is bad? Edit: re-read the article and it seems that case would still work. But deciding the user is bad based on some arbitrary metric still seems like something one should have care with. Edit: downvotes confirm web kids don't give a damn that you use flawed heuristics to determine someone is "bad". Some will copy paste this into their project and the unfortunate user who happens to have a less common configuration will be screwed.

oliv__ 9y ago
No, downvotes confirm that you are stigmatizing a good chunk of the readership of this website who will not care to read further into your comment than the first 7 words.
- Spooky23 9y ago
  He lacks tact, but has a point.
  - asveikau 9y ago
    Thanks. Just to clarify, I believe that if someone posts code snippets like this, purporting to do thing X by doing roundabout and overly simplistic thing Y, you really have to own it and assume tons of responsibility for misleading fellow developers who may be unaware of possible edge cases. When somebody copy-pastes a quick solution and it doesn't work for literally everybody, that translates to real people getting a bad deal when they hit your bugs, something we should all as responsible developers seek to avoid. And yeah, I will not waste time being tactful when describing this situation, because I have seen it before and it kind of sucks.

PascLeRasc 9y ago

I don't want to start an argument here, but can someone explain why web scraping is considered malicious?

sillysaurus3 9y ago
Many websites have ToS's that forbid scraping.
- deleted 9y ago
  [deleted]
zeta0134 9y ago
I don't believe that responsible web scraping is malicious, but my belief relies on the assumption that the web is meant to be open. That is, information put on the internet that is publicly available is considered free to access, store, and later retransmit. The original web was designed for researchers to share their work, while the modern internet built on top of that platform has other moral systems that don't necessarily agree. Anyway, on a purely technical level, scraping of publicly available content isn't inherently bad unless you're asked to stop, or are scraping so quickly as to cause a service disruption by tying up the target systems. There is nothing malicious about generating normal traffic at the rate of a regular user. The animosity arises from what you plan to do with the data, and whether the entity you're scraping agrees with your usage.
- PascLeRasc 9y ago
  Thanks, that's what seems obvious to me too that it's just public data and it's possible to collect it without overwhelming the server with requests. I just don't get why someone wouldn't want you to look at their website.

fiatjaf 9y ago

Isn't it possible to detect a bot by tracking some events like random mouse moving, scrolling, clicking etc.? Why weren't these kinds of detection tried in place of captchas, for example?

kuschku 9y ago
Because they are easily faked. Google’s current captcha system does track a few of these, but it mostly takes your browsing history, and, if that seems normal, will accept you. I’ve run a few IRC bots that allowed people to submit Google searches, and would return the first resulting link. They also fetch any link mentioned in IRC channels, execute the JS, and after a timeout of 400ms respond with the current page title. Both combined – a normal search history, reading a few hundred pages and videos a day per user – apparently are enough that they seem "human", and can pass NoCaptcha.

tomatsu 9y ago

> var body = document.getElementsByTagName("body")[0]; You can just use document.body. I also suggest to use a data URL instead. E.g. "data:," is an empty plain text file, which, as you can imagine, won't be interpreted as a valid image. let image = new Image(); image.onerror = () => { console.log(image.width); // 0 -> headless }; document.body.appendChild(image); image.src = 'data:,'; > In case of a vanilla Chrome, the image has a width and height that depends on the zoom of the browser The zoom doesn't affect this. It's always in CSS "pixels".

askvictor 9y ago

All of these could quite easily be overcome by compiling your own headless chrome. It wouldn't surprise me if there is a fork to this effect soon.

assafmo 9y ago

"... a headless browser can also be used to automate malicious tasks. The most common cases are web scraping... " Since when web scraping considered malicious? Companies like Google are doing billions because they use web scraping.

DannyDaemonic 9y ago

I'd be willing to bet that missing image size variance is more of a bug or oversight, and is something that will be fixed.

albertgonz14 9y ago

Need a professional hacker that is dedicated to Internet security and all things that deal with digital security ?,..I specialize in most fields of Apple ,Andriod,Microsoft,partners sincerity,employee's honesty,recover your email password,social networks,university grades changing/admin(staff) account hack-access/password,facebook,Instagram,bbm,Skype,snapchat,zoosk,match,eharmony,various blogs,clearing of criminal records,hack gmail accounts,yahoomail,hotmail,aol..gain access to bank accounts several other platforms that constantly run test on different networks and operating system to ensure complete security and functionality .all the services we offer are LEGAL..contact email:Albertgonzalez11960@gmail.com ,whatsapp:+15157170625

hossbeast 9y ago

"Beyond the two harmless use cases given previously, a headless browser can also be used to automate malicious tasks. The most common cases are web scraping, increase advertisement impressions or look for vulnerabilities on a website." Cheating an advertiser I'll grant you, but the other two are 100% legitimate.

josteink 9y ago

> Beyond the two harmless use cases given previously, a headless browser can also be used to automate malicious tasks. The most common cases are web scraping I guess I disagree with the premise of this article. How is web scraping fundamental malicious? What rights/expectations can you have that a publicly accessible website you create must be used by humans only?

sumedh 9y ago
It puts a load on you server when bots go wild on your site which in turn affects the experience of legitimate human users of the site.

tscs37 9y ago

I do hope that these methods get patched, I tend to archive my bookmark collection with chrome headless to prevent loosing content when such a site goes offline. I hate it when a website requires me to play special snowflake to scrape them for this purpose.

jdc0589 9y ago

dumb question from someone who's written a ton of scrapers and scraping based "products" for fun: at what point does it make more sense for companies to just start offering open APIs or data exports? Obviously it would never make sense for a company who's value IS their data, but for retail platforms, auction sites, forum platforms, etc... that have a scraper problem, it seems like just providing their useful data through a more controlled, and optimized, avenue could be worth it. The answer is probably "never", it's just something that comes to mind sometimes.

megamindbrian 9y ago

Can you guys shut up already?