6 ms·

Detecting PhantomJS-based visitors

116 points by bentlegen 12y ago

deloschang 12y ago
Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?
- dsp1234 12y ago
  The first thing that comes to mind is that Selenium doesn't generate mouse move events when doing something like clicking two different buttons on a page, or when testing something like hover. So I would definitely concentrate on user-generated events, assuming that it's not easy to detect on the backend (via something like user-agent).
  - deleted 12y ago
    [deleted]
  - Torgo 12y ago
    Pass a user agent saying you're a touchscreen browser.
andrenotgiant 12y ago
The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA. For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns. I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.
- weinzierl 12y ago
  My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list. When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time. Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.
  - aselzer 12y ago
    I also think it doesn't have to be that complex. I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha. Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened. The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else. If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.
- curiously 12y ago
  I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly. I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.
  - andrenotgiant 12y ago
    There has to be some calibration or customization beyond IP. Traffic from University IP vs my home looks very different.
- snowwrestler 12y ago
  This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.
joshstrange 12y ago
With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?
- jsoverson 12y ago
  Things like phantomjs, not phantomjs. PhantomJS is often used to automate sites against the TOS.
- nkozyra 12y ago
  I doubt any serious search engine is literally using PhantomJS, so I wouldn't see this as a huge problem. Google is in all likelihood using V8 directly and is explicit about who it is. In fact, most legitimate search engine spiders are.
  - joshstrange 12y ago
    I agree, I was speaking more on trying to block crawlers that look like browsers in general, sorry that wasn't more clear.
homakov 12y ago
These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"
- curiously 12y ago
  I agree with homakov. The article itself highlights defenses which are ridiculously naive and easy to bypass. Worst case, you don't use phantomjs, most of these tricks to block will be useless.
- bentlegen 12y ago
  A week is a subset of eventually, so technically the statement is true. On a serious note - the article mentions repeatedly that many of the techniques can be trivially defeated. The main purpose of the article was to highlight some interesting, unique properties of PhantomJS that most people are not aware of (e.g. stack trace analysis).
  - homakov 12y ago
    I agree that highlighted properties are interesting, but if we're talking about practical usage of these techniques it makes almost no sense (benefits to efforts ratio is too low)
    - speedy_go 12y ago
      Efforts to detect a script kiddies (or anyone who didn't bother heavily modify PhantomJS/QtWebKit/WebKit) by looking at properties mentioned in the article are really nothing compared to other approaches people use these days.
- discreditable 12y ago
  > These techniques are extremely easy to bypass. Especially considering nowadays you can automate IE with PowerShell [1] . I've used it myself to automate account creation on a legacy system at work. [1] http://www.youdidwhatwithtsql.com/automating-internet-explorer-with-powershell/467/ http://www.youdidwhatwithtsql.com/automating-internet-explor...
dpweb 12y ago
I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see your ads or sign up for your deal, and just inflate your traffic numbers (unless you find that desirable). There are a number of ways to try to detect and trick them out. I've done some unusual things to attempt to detect them, finding people coming thru proxies as well. Sure, these simple methods can be defeated by an adversary, but at least judging from my experience, even simple methods you're going to catch some of them. Some hackers ain't that smart. A little surpised Cloudflare for instance wouldn't offer this (PhantomJS blocking), if they don't already. I'm all for hacking, but a site owner you should have the right to block bots if you so choose.
- jalfresi 12y ago
  If you don't want it crawled, don't put it on the public internet. You're just participating in an arms race.
logn 12y ago
I'm currently writing a Java-based headless browser, https://github.com/MachinePublishers/jBrowserDriver https://github.com/MachinePublishers/jBrowserDriver Beware it's Affero GPL. It implements most of the Selenium WebDriver APIs. Currently it's a work in progress especially in regard to persistent tracking. But it's capable of addressing many issues raised in the article. As a bonus, for Java users it's a lot faster than GhostDriver at least. Header ordering: https://github.com/MachinePublishers/jBrowserDriver/blob/master/browser/src/com/machinepublishers/jbrowserdriver/Settings.java https://github.com/MachinePublishers/jBrowserDriver/blob/mas... DOM objects (user-agent, navigator, Canvas, Date) are addressed by injecting the response content: https://github.com/MachinePublishers/jBrowserDriver/blob/master/browser/src/com/machinepublishers/jbrowserdriver/StreamHandler.java https://github.com/MachinePublishers/jBrowserDriver/blob/mas... JS Engine: about 1 year newer than PhantomJS. Java ships with Qt5 WebKit. I'll have some significant updates in the next couple of days surrounding all these issues.
- jacquesm 12y ago
  That's a super piece of work! Thank you, I'll probably become one of your users.
jdangu 12y ago
Companies like WhiteOps and formerly Spider.io (now Google) are doing this as-a-service to detect useragent spoofing for all browsers, with hundreds of data points (mostly to detect ad fraud)

116 points by bentlegen 12y ago

Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?

dsp1234 12y ago
The first thing that comes to mind is that Selenium doesn't generate mouse move events when doing something like clicking two different buttons on a page, or when testing something like hover. So I would definitely concentrate on user-generated events, assuming that it's not easy to detect on the backend (via something like user-agent).
- deleted 12y ago
  [deleted]
- Torgo 12y ago
  Pass a user agent saying you're a touchscreen browser.

andrenotgiant 12y ago

The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA. For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns. I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.

weinzierl 12y ago
My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list. When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time. Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.
- aselzer 12y ago
  I also think it doesn't have to be that complex. I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha. Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened. The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else. If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.
curiously 12y ago
I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly. I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.
- andrenotgiant 12y ago
  There has to be some calibration or customization beyond IP. Traffic from University IP vs my home looks very different.
snowwrestler 12y ago
This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.

joshstrange 12y ago

With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?

jsoverson 12y ago
Things like phantomjs, not phantomjs. PhantomJS is often used to automate sites against the TOS.
nkozyra 12y ago
I doubt any serious search engine is literally using PhantomJS, so I wouldn't see this as a huge problem. Google is in all likelihood using V8 directly and is explicit about who it is. In fact, most legitimate search engine spiders are.
- joshstrange 12y ago
  I agree, I was speaking more on trying to block crawlers that look like browsers in general, sorry that wasn't more clear.

homakov 12y ago

These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"

curiously 12y ago
I agree with homakov. The article itself highlights defenses which are ridiculously naive and easy to bypass. Worst case, you don't use phantomjs, most of these tricks to block will be useless.
bentlegen 12y ago
A week is a subset of eventually, so technically the statement is true. On a serious note - the article mentions repeatedly that many of the techniques can be trivially defeated. The main purpose of the article was to highlight some interesting, unique properties of PhantomJS that most people are not aware of (e.g. stack trace analysis).
- homakov 12y ago
  I agree that highlighted properties are interesting, but if we're talking about practical usage of these techniques it makes almost no sense (benefits to efforts ratio is too low)
  - speedy_go 12y ago
    Efforts to detect a script kiddies (or anyone who didn't bother heavily modify PhantomJS/QtWebKit/WebKit) by looking at properties mentioned in the article are really nothing compared to other approaches people use these days.
discreditable 12y ago
> These techniques are extremely easy to bypass. Especially considering nowadays you can automate IE with PowerShell [1] . I've used it myself to automate account creation on a legacy system at work. [1] http://www.youdidwhatwithtsql.com/automating-internet-explorer-with-powershell/467/ http://www.youdidwhatwithtsql.com/automating-internet-explor...

dpweb 12y ago

I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see your ads or sign up for your deal, and just inflate your traffic numbers (unless you find that desirable). There are a number of ways to try to detect and trick them out. I've done some unusual things to attempt to detect them, finding people coming thru proxies as well. Sure, these simple methods can be defeated by an adversary, but at least judging from my experience, even simple methods you're going to catch some of them. Some hackers ain't that smart. A little surpised Cloudflare for instance wouldn't offer this (PhantomJS blocking), if they don't already. I'm all for hacking, but a site owner you should have the right to block bots if you so choose.

jalfresi 12y ago
If you don't want it crawled, don't put it on the public internet. You're just participating in an arms race.

logn 12y ago

I'm currently writing a Java-based headless browser, https://github.com/MachinePublishers/jBrowserDriver https://github.com/MachinePublishers/jBrowserDriver Beware it's Affero GPL. It implements most of the Selenium WebDriver APIs. Currently it's a work in progress especially in regard to persistent tracking. But it's capable of addressing many issues raised in the article. As a bonus, for Java users it's a lot faster than GhostDriver at least. Header ordering: https://github.com/MachinePublishers/jBrowserDriver/blob/master/browser/src/com/machinepublishers/jbrowserdriver/Settings.java https://github.com/MachinePublishers/jBrowserDriver/blob/mas... DOM objects (user-agent, navigator, Canvas, Date) are addressed by injecting the response content: https://github.com/MachinePublishers/jBrowserDriver/blob/master/browser/src/com/machinepublishers/jbrowserdriver/StreamHandler.java https://github.com/MachinePublishers/jBrowserDriver/blob/mas... JS Engine: about 1 year newer than PhantomJS. Java ships with Qt5 WebKit. I'll have some significant updates in the next couple of days surrounding all these issues.

jacquesm 12y ago
That's a super piece of work! Thank you, I'll probably become one of your users.

jdangu 12y ago

Companies like WhiteOps and formerly Spider.io (now Google) are doing this as-a-service to detect useragent spoofing for all browsers, with hundreds of data points (mostly to detect ad fraud)