4 ms·

The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaSc

The other day I've made a Chrome extension for scrapping a protected website. It worked wonderfully, as it simulated a normal user session, bypassing the JavaScript protections the website has. You can also run such scripts with a headless browser for full automation, PhantomJS being an obvious choice. You really can't protect against this unless you start making the experience of regular visitors much worse.

amelius 9y ago
But your crawler must be very slow then.
- pmx 9y ago
  They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.
  - alexcnwy 9y ago
    Plus very easy to parallelize
    - amelius 9y ago
      Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).
inglor 9y ago
Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).
- pbhjpbhj 9y ago
  IF you throttle the connection then wouldn't download time and network latency, etc., give you a better random?
  - StavrosK 9y ago
    You can generally tell because of the distribution, which tends to show up after lots of data points.
- bad_user 9y ago
  The issue here isn't that it can't be done, the problem being one of false positives. By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare. And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular, in which case you'll piss of enough users as to receive bad reviews. Basically you can't afford to do this, unless you're Facebook or Google, and you have to then wonder why Facebook or Google do not deploy such protections So going back to my main point, of course it's possible, but the experience for users gets significantly worse such that you won't want to do it.
  - inglor 9y ago
    I've found Cloudflare's solution to be very inadequate compared to Distil or Incapsula. Both Distil and Incapsula _did_ have a small number of false positives (showing a captcha to users). We did have to write some code to overcome that.
- laughfactory 9y ago
  Yeah, but if you can use ML to pick out the differences between real users and bots then you can also use ML to mimic real user behavior. Both sides spend a lot of time and money trying to overcome each other, and end state is the same it was at the beginning: near total access to the data, at least for those sophisticated enough.
winkv 9y ago
Interesting, i am facing problem with this approach.There is a site which pops up alert box for username and password and i can't seem to switch to it with chrome web extension.With selenium it is as simple as driver.switchTo().alert(); have you ever faced this problem? by any chance you know how to fix it? I am trying to accomplish the same thing as described in this question in SO https://sqa.stackexchange.com/questions/20710/not-able-to-use-http-basic-auth-popup-in-selenium-test-case-using-firefox-on-ubu https://sqa.stackexchange.com/questions/20710/not-able-to-us...

amelius 9y ago

But your crawler must be very slow then.

pmx 9y ago
They generally are fairly slow but it doesn't matter because you just set it and leave it to it until it has all the data you want, leaving you to get on with other things.
- alexcnwy 9y ago
  Plus very easy to parallelize
  - amelius 9y ago
    Unless the websites work together to protect themselves against crawlers that work that way (e.g., rejecting requests to different hosts from the same origin within a time interval).

inglor 9y ago

Sure you can protect against this - there are several companies that use machine learning to spot small differences between selenium and real users (mouse delays etc). For example, it might detect that a mouse click is dispatched at exact intervals (and block it). To which you'd think "I'll just add Math.random() * 2000" which it'll easily detect as well. It's _definitely_ doable, but it's not as trivial as recording a selenium macro. (Not to mention these tools look for selenium presence and extensions anyway).

pbhjpbhj 9y ago
IF you throttle the connection then wouldn't download time and network latency, etc., give you a better random?
- StavrosK 9y ago
  You can generally tell because of the distribution, which tends to show up after lots of data points.
bad_user 9y ago
The issue here isn't that it can't be done, the problem being one of false positives. By doing this you must accept that a certain percentage of legitimitate users, which can be quite significant, will be blocked. In case you're wondering, yes, this does happen with solutions such as Cloudflare. And at that point, either your website isn't popular, in which case you can't afford to lose users, or it is very popular, in which case you'll piss of enough users as to receive bad reviews. Basically you can't afford to do this, unless you're Facebook or Google, and you have to then wonder why Facebook or Google do not deploy such protections So going back to my main point, of course it's possible, but the experience for users gets significantly worse such that you won't want to do it.
- inglor 9y ago
  I've found Cloudflare's solution to be very inadequate compared to Distil or Incapsula. Both Distil and Incapsula _did_ have a small number of false positives (showing a captcha to users). We did have to write some code to overcome that.
laughfactory 9y ago
Yeah, but if you can use ML to pick out the differences between real users and bots then you can also use ML to mimic real user behavior. Both sides spend a lot of time and money trying to overcome each other, and end state is the same it was at the beginning: near total access to the data, at least for those sophisticated enough.

winkv 9y ago

Interesting, i am facing problem with this approach.There is a site which pops up alert box for username and password and i can't seem to switch to it with chrome web extension.With selenium it is as simple as driver.switchTo().alert(); have you ever faced this problem? by any chance you know how to fix it? I am trying to accomplish the same thing as described in this question in SO https://sqa.stackexchange.com/questions/20710/not-able-to-use-http-basic-auth-popup-in-selenium-test-case-using-firefox-on-ubu https://sqa.stackexchange.com/questions/20710/not-able-to-us...