9 ms·

Your solutions in detecting Chrome headless is good. But someone who really wants to do web scraping or anything similar will use a real browser like Firefox o

Your solutions in detecting Chrome headless is good. But someone who really wants to do web scraping or anything similar will use a real browser like Firefox or Chrome run it through xvfb and control it using webdriver and maybe expose it through an API. I find these to be almost undetectable.. The only way you can mitigate this is to do more interesting mitigation techniques. Liie IP detection, Captchas, etc. edit: when I say real browser, I mean running the full browser process including extensions etc.

dsacco 9y ago
Interesting. I've done a lot of scraping and I never did this. Do you have a treatise that explains how it's done? I've actually been experimenting with headless Chrome recently because I like it more than PhantomJS. Practically speaking I don't think IP detection is useful at all these days, and the only Captcha that can't be bypassed is Google's most recent version. The much more successful anti-scraping tactic is to use a sophisticated reverse proxy that analyzes all requests to identify patterns and unusual behavior, indiscriminate from the IP source (because any large scale scraping will come from many IPs anyway).
- heipei 9y ago
  If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
- IgorPartola 9y ago
  IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?
  - heipei 9y ago
    There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/
    - ant6n 9y ago
      What about tor.
      - heipei 9y ago
        TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
        crtasm 9y ago
        I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
      - dsacco 9y ago
        Tor would introduce pretty intolerable latency into a professional data crawling setup.
    - ficklepickle 9y ago
      I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...
  - tyingq 9y ago
    >Having said that, just let them scrape? Depends on who you are. Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Letting them scrape with no limits is costly.
    - dsacco 9y ago
      > Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Yes, and they make it very cumbersome to discourage it. I've successfully written scrapers for airlines. It's significantly more difficult than crawling other websites for a few reasons: 1. Session management is wacky - they really like to manage state entirely through cookies, and you typically need to visit a specific set of pages in a specific sequence before you can access the resources you want, like the number of seats available or their prices. 2. Sessions have time limits because anyone who looks at the seats initiates a "soft" reservation on them (this works in a similar way for theatre, concert and movie seating). 3. You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing (which, given the type of HTML you encounter, can be hell).
      - tyingq 9y ago
        >You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.
        dsacco 9y ago
        Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
    - IgorPartola 9y ago
      And why is that a good thing for them? I mean if they provided an API to get their pricing, wouldn't it be in their interest? Does anyone buy airline tickets from the airline website ever? I don't. I use sites like Expedia and Priceline. And if an airline is not listed or has shitty prices, I don't buy it. Wouldn't it be in their interest to be listed on there? Moreover, wouldn't they want the other companies to have similar API's to have dynamic pricing? I get that at one point not having this info out there was a good thing for them. But now the car is out of the bag. You can't keep pretending that booking sites don't exist.
  - jalfresi 9y ago
    There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
    - deleted 9y ago
      [deleted]
    - deleted 9y ago
      [deleted]
  - jalfresi 9y ago
    There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
- skinnymuch 9y ago
  I'm thinking OP meant if you want to never be detected, but like you said, you usually never get detected anyway. Bypassing captcha automatically or with captcha bypass services where you pay per captcha completion?
- pebers 9y ago
  It's not terribly hard using Selenium; you can use one of their Docker images (e.g. selenium/standalone-chrome) and connect to that using Webdriver. Having said that, it has flaws compared to headless Chrome (more moving parts, terrible security, extra memory usage / dependencies for the JVM to run Selenium) so unless that camouflage is proving crucial, I'd avoid it where possible. I've recently migrated a project to headless Chrome and definitely prefer it.
  - taeric 9y ago
    I'm curious on what makes the security worse. The rest, I can see. Though, I have to confess I have been more critical of the whole headless browser binary push than I'd care to acknowledge. I just feel like this is solving a problem that didn't actually exist anymore. Selenium has worked solidly for quite a while. And, you almost certainly should test in multiple browsers/scenarios.
    - pebers 9y ago
      The main security issues I know of so far are: - Neither Selenium nor Webdriver (at least the Python client, but I assume others too) support HTTPS at all. - By default it opens browsers configured to accept any SSL certificate. I can see why that's useful for local testing, but it's a terrible choice to default to. - It logs way too much. Logs every keypress sent to it, including passwords. At least the latter two are fixable, but not trivially so, and better defaults with easy client-side options to opt into insecure mode would have been much better. The former is not fixable without patching both the client and server and in 2017 it's pretty poor to not even have an option for.
      - taeric 9y ago
        Odd, I'm pretty sure both of the first two are wrong. We used to have to make sure we had legit certs for selenium to work with https. That or configure the Firefox profile to have already accepted the self signed one. And, really, the first two points are clearly contradictory. I'm guessing I just misunderstand what you mean? Do you mean the communication between the driver? I'm curious why ssl would be important there? Should just be done locally and a standard ssh tunnel can help with any remote encryption you might want.
- kuschku 9y ago
  > and the only Captcha that can't be bypassed is Google's most recent version That can also be easily bypassed (I’ve done that, without actually planning to do so) if you can make your behaviour seem completely human. So if you have a bot that can browse in a way that seems statistically human you can actually get around that. You can still scrape the same datasets – you just need to keep instances of bots, with all cookies around, and have them browse in certain ways. Classify what category a site might likely belong to, put them into buckets of queues, and have bots pull from queues they’re likely to browse, or search for certain terms they’re likely to search for. In my case, this all happened by accidents – I had IRC bots for a few channels, each kept a perpetual session and would visit every site that was linked (to get the title) and would be able to search Google by scraping. One day I was accessing a bot remotely, and told it to access a page that was NoCaptcha protected (because the site wasn’t working on my home system), yet it passed the captchas perfectly fine. Tried a few more times, always worked. So I tried figuring out why it worked.
  - _e 9y ago
    Were the irc bots able to render javascript or were the bots using a cli lib like python requests? I thought that a client which doesnt run javascript would be a huge red flag.
    - kuschku 9y ago
      They used Firefox with off-screen rendering, as otherwise they wouldn’t be able to get the title of many pages – even many blogspot blogs can’t be read without JS anymore.
ivanhoe 9y ago
Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.
- jordanpg 9y ago
  In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text
  - dsacco 9y ago
    To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
    - ivanhoe 9y ago
      The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
aleem 9y ago
The simplest and most effective trick is to have a link that is not visible to humans but only to crawlers. When an IP ends up on that link you know it's a crawler. You can then proceed to block it. A honeypot in other words. Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.
- threatofrain 9y ago
  That sounds like a great trick, but I'd want a solution against someone getting large swathes of IP's banned, because the bot crawled my site and posted the link somewhere.
- chongli 9y ago
  Be careful with that. Users with screen readers may fall into your trap.
  - stordoff 9y ago
    Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window? Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
- downandout 9y ago
  I do something similar to this on PPC campaigns, but not for the purpose of blocking IPs. Bots tend to come in waves from a given site. If more than X% of clients from a given referrer wind up on the fake link (or other techniques show that the clients are bots), my code can automatically suspend the display of ads on the offending site for a period of time. If it continually happens, then the site's owner is the likely culprit, so I have a threshold at which that site is automatically and permanently blacklisted from all of my campaigns. I have found that advanced bot mitigation is the single most significant determining factor of PPC ROI, which is a sad commentary on the current state of the paid advertising ecosystem.
amelius 9y ago
Do you also use OCR? Because I guess you could defeat webscrapers by making the HTML structurally totally different from what is visually present in the browser window.
- kbutler 9y ago
  As has been mentioned elsewhere in the thread, you need to be careful about accessibility and screen reader access.
baybal2 9y ago
Yes, it will do nothing against sophisticated botters. It has been a standard tactic to run browsers in purpose built VMs. The last bastion were smarter, more disruptive captchas and rendering performance profiling, but even that got grinded by now. The situation is even more grave with mobile ads as you have zero opportunity to shove captchas, or rely on single unique per-system ID or cookie that was vetted outside of embedded webkit environment.
feelin_googley 9y ago
Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
- feelin_googley 9y ago
  One can also use a microcontroller such as Teensy to script timed keyboard and mouse action. Demonstrations have usually been by security researchers, but these tiny boards can be used wherever one wants to avoid the labor of repetitive navigating, typing and mouse clicking. http://samy.pl/usbdriveby http://samy.pl/usbdriveby
kccqzy 9y ago
I've done similar detections. Basically just use keyboard dynamics: by measuring the precise timing of keystrokes (key down, key up) you can tell if it is likely typed by a user manually. I haven't done it but I think mouse movements could be used too. Add onMouseMove event listeners all over the page and see how they are triggered.
- notgood 9y ago
  Yeah, that's why I use Math.random() * 5 and similar stuff in my scrapers. A lot of people access the web through smartphones so only allowing people with mouse movements seems a bit overkill.
feelin_googley 9y ago
Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....
yes_or_gnome 9y ago
Safari has navigator.webdriver, and others will soon follow. The idea was first introduced as a Fx bug, but hasn't gone any where. https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 There's a webdriver protocol mailing list with a long thread that describes how it should be implemented and how people could avoid it; ex. recompile without the feature. I just can't find it right this minute.
deleted 9y ago
[deleted]
tehlike 9y ago
There is a sevice for solving captchas. Death by captcha is one
tehlike 9y ago
There is a sevice for solving captchas. Death by captcha is one

dsacco 9y ago

Interesting. I've done a lot of scraping and I never did this. Do you have a treatise that explains how it's done? I've actually been experimenting with headless Chrome recently because I like it more than PhantomJS. Practically speaking I don't think IP detection is useful at all these days, and the only Captcha that can't be bypassed is Google's most recent version. The much more successful anti-scraping tactic is to use a sophisticated reverse proxy that analyzes all requests to identify patterns and unusual behavior, indiscriminate from the IP source (because any large scale scraping will come from many IPs anyway).

heipei 9y ago
If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
IgorPartola 9y ago
IP detection, especially IPv4, should still work fine because the cost to get a new IP will often outweigh the benefit of the scraping. Having said that, just let them scrape?
- heipei 9y ago
  There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic will exit with actual users on consumer ISP lines: http://hola.org/ http://hola.org/
  - ant6n 9y ago
    What about tor.
    - heipei 9y ago
      TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes (https://torstatus.blutmagie.de/ https://torstatus.blutmagie.de/), and unless you exit via a non-listed one you're trivially identifiable. Lastly, TOR is rather slow, which prohibits using for large-scale scraping tasks. Plus you get switched around different exits and countries, which might break your scraping logic.
      - crtasm 9y ago
        I believe you could use the ExitNodes option to only use a specific exit. There's no such thing as an unlisted exit node, only unlisted entry nodes (bridges). I agree that Tor isn't a good choice for scraping.
    - dsacco 9y ago
      Tor would introduce pretty intolerable latency into a professional data crawling setup.
  - ficklepickle 9y ago
    I'd say don't use Hola. They've had some issues, like selling their users bandwidth for botnets[1]. If you do some googling, you will see many reasons not to use that extension. [1]http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-selling-users-bandwidth http://www.theverge.com/2015/5/29/8685251/hola-vpn-botnet-se...
- tyingq 9y ago
  >Having said that, just let them scrape? Depends on who you are. Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Letting them scrape with no limits is costly.
  - dsacco 9y ago
    > Hotel and airline websites, for example, are pummeled with scrapers wanting pricing and availability info. Yes, and they make it very cumbersome to discourage it. I've successfully written scrapers for airlines. It's significantly more difficult than crawling other websites for a few reasons: 1. Session management is wacky - they really like to manage state entirely through cookies, and you typically need to visit a specific set of pages in a specific sequence before you can access the resources you want, like the number of seats available or their prices. 2. Sessions have time limits because anyone who looks at the seats initiates a "soft" reservation on them (this works in a similar way for theatre, concert and movie seating). 3. You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing (which, given the type of HTML you encounter, can be hell).
    - tyingq 9y ago
      >You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.
      - dsacco 9y ago
        Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
  - IgorPartola 9y ago
    And why is that a good thing for them? I mean if they provided an API to get their pricing, wouldn't it be in their interest? Does anyone buy airline tickets from the airline website ever? I don't. I use sites like Expedia and Priceline. And if an airline is not listed or has shitty prices, I don't buy it. Wouldn't it be in their interest to be listed on there? Moreover, wouldn't they want the other companies to have similar API's to have dynamic pricing? I get that at one point not having this info out there was a good thing for them. But now the car is out of the bag. You can't keep pretending that booking sites don't exist.
- jalfresi 9y ago
  There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
  - deleted 9y ago
    [deleted]
  - deleted 9y ago
    [deleted]
- jalfresi 9y ago
  There exists an entire industry that provide rotating IPs across fleets of proxies for very cheap prices. Proxy services like this make it pretty much impossible to prevent scraping.
skinnymuch 9y ago
I'm thinking OP meant if you want to never be detected, but like you said, you usually never get detected anyway. Bypassing captcha automatically or with captcha bypass services where you pay per captcha completion?
pebers 9y ago
It's not terribly hard using Selenium; you can use one of their Docker images (e.g. selenium/standalone-chrome) and connect to that using Webdriver. Having said that, it has flaws compared to headless Chrome (more moving parts, terrible security, extra memory usage / dependencies for the JVM to run Selenium) so unless that camouflage is proving crucial, I'd avoid it where possible. I've recently migrated a project to headless Chrome and definitely prefer it.
- taeric 9y ago
  I'm curious on what makes the security worse. The rest, I can see. Though, I have to confess I have been more critical of the whole headless browser binary push than I'd care to acknowledge. I just feel like this is solving a problem that didn't actually exist anymore. Selenium has worked solidly for quite a while. And, you almost certainly should test in multiple browsers/scenarios.
  - pebers 9y ago
    The main security issues I know of so far are: - Neither Selenium nor Webdriver (at least the Python client, but I assume others too) support HTTPS at all. - By default it opens browsers configured to accept any SSL certificate. I can see why that's useful for local testing, but it's a terrible choice to default to. - It logs way too much. Logs every keypress sent to it, including passwords. At least the latter two are fixable, but not trivially so, and better defaults with easy client-side options to opt into insecure mode would have been much better. The former is not fixable without patching both the client and server and in 2017 it's pretty poor to not even have an option for.
    - taeric 9y ago
      Odd, I'm pretty sure both of the first two are wrong. We used to have to make sure we had legit certs for selenium to work with https. That or configure the Firefox profile to have already accepted the self signed one. And, really, the first two points are clearly contradictory. I'm guessing I just misunderstand what you mean? Do you mean the communication between the driver? I'm curious why ssl would be important there? Should just be done locally and a standard ssh tunnel can help with any remote encryption you might want.
kuschku 9y ago
> and the only Captcha that can't be bypassed is Google's most recent version That can also be easily bypassed (I’ve done that, without actually planning to do so) if you can make your behaviour seem completely human. So if you have a bot that can browse in a way that seems statistically human you can actually get around that. You can still scrape the same datasets – you just need to keep instances of bots, with all cookies around, and have them browse in certain ways. Classify what category a site might likely belong to, put them into buckets of queues, and have bots pull from queues they’re likely to browse, or search for certain terms they’re likely to search for. In my case, this all happened by accidents – I had IRC bots for a few channels, each kept a perpetual session and would visit every site that was linked (to get the title) and would be able to search Google by scraping. One day I was accessing a bot remotely, and told it to access a page that was NoCaptcha protected (because the site wasn’t working on my home system), yet it passed the captchas perfectly fine. Tried a few more times, always worked. So I tried figuring out why it worked.
- _e 9y ago
  Were the irc bots able to render javascript or were the bots using a cli lib like python requests? I thought that a client which doesnt run javascript would be a huge red flag.
  - kuschku 9y ago
    They used Firefox with off-screen rendering, as otherwise they wouldn’t be able to get the title of many pages – even many blogspot blogs can’t be read without JS anymore.

ivanhoe 9y ago

Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that. Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.

jordanpg 9y ago
In addition to complicated, making things unpredictable makes things very difficult for programmatic scraping: * machine-generated HTML and CSS identifiers * randomly inserted, unused HTML elements * images instead of text
- dsacco 9y ago
  To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it. Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is. And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags https://stackoverflow.com/questions/1732348/regex-match-open...
  - ivanhoe 9y ago
    The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.

aleem 9y ago

The simplest and most effective trick is to have a link that is not visible to humans but only to crawlers. When an IP ends up on that link you know it's a crawler. You can then proceed to block it. A honeypot in other words. Typically display none will do but you can also have white on white text with no tabindex or off screen absolute positioning, etc.

threatofrain 9y ago
That sounds like a great trick, but I'd want a solution against someone getting large swathes of IP's banned, because the bot crawled my site and posted the link somewhere.
chongli 9y ago
Be careful with that. Users with screen readers may fall into your trap.
- stordoff 9y ago
  Could you have the content of the page explain that its a bot trap (so screen reader users know its not worth visiting) and only block if, say, an IP visits it multiple times in a short window? Wouldn't block specialised scrapers as they would know to avoid that URL/link (though they'd probably work it out anyway), but would still limit more broad crawlers.
downandout 9y ago
I do something similar to this on PPC campaigns, but not for the purpose of blocking IPs. Bots tend to come in waves from a given site. If more than X% of clients from a given referrer wind up on the fake link (or other techniques show that the clients are bots), my code can automatically suspend the display of ads on the offending site for a period of time. If it continually happens, then the site's owner is the likely culprit, so I have a threshold at which that site is automatically and permanently blacklisted from all of my campaigns. I have found that advanced bot mitigation is the single most significant determining factor of PPC ROI, which is a sad commentary on the current state of the paid advertising ecosystem.

amelius 9y ago

Do you also use OCR? Because I guess you could defeat webscrapers by making the HTML structurally totally different from what is visually present in the browser window.

kbutler 9y ago
As has been mentioned elsewhere in the thread, you need to be careful about accessibility and screen reader access.

baybal2 9y ago

Yes, it will do nothing against sophisticated botters. It has been a standard tactic to run browsers in purpose built VMs. The last bastion were smarter, more disruptive captchas and rendering performance profiling, but even that got grinded by now. The situation is even more grave with mobile ads as you have zero opportunity to shove captchas, or rely on single unique per-system ID or cookie that was vetted outside of embedded webkit environment.

feelin_googley 9y ago

Long time ago we used Crowbar with xvfb. Nice to see the same method still works. http://web.archive.org/web/20121127001253/http://simile.mit.edu:80/wiki/Crowbar http://web.archive.org/web/20121127001253/http://simile.mit....

feelin_googley 9y ago
One can also use a microcontroller such as Teensy to script timed keyboard and mouse action. Demonstrations have usually been by security researchers, but these tiny boards can be used wherever one wants to avoid the labor of repetitive navigating, typing and mouse clicking. http://samy.pl/usbdriveby http://samy.pl/usbdriveby

kccqzy 9y ago

I've done similar detections. Basically just use keyboard dynamics: by measuring the precise timing of keystrokes (key down, key up) you can tell if it is likely typed by a user manually. I haven't done it but I think mouse movements could be used too. Add onMouseMove event listeners all over the page and see how they are triggered.

notgood 9y ago
Yeah, that's why I use Math.random() * 5 and similar stuff in my scrapers. A lot of people access the web through smartphones so only allowing people with mouse movements seems a bit overkill.

feelin_googley 9y ago

yes_or_gnome 9y ago

Safari has navigator.webdriver, and others will soon follow. The idea was first introduced as a Fx bug, but hasn't gone any where. https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 https://bugzilla.mozilla.org/show_bug.cgi?id=1169290 There's a webdriver protocol mailing list with a long thread that describes how it should be implemented and how people could avoid it; ex. recompile without the feature. I just can't find it right this minute.

deleted 9y ago

[deleted]

tehlike 9y ago

There is a sevice for solving captchas. Death by captcha is one

tehlike 9y ago

There is a sevice for solving captchas. Death by captcha is one