4 ms·

"That’s when it becomes impossible. You can come up with whatever tests you want, but any dedicated web scraper can easily get around them." As long as the log

"That’s when it becomes impossible. You can come up with whatever tests you want, but any dedicated web scraper can easily get around them." As long as the logic is hidden from the scrapers, i.e. not running in a web browser, scrapers are at a disadvantage. They don't have the data about the users that websites have. And even something as simple as Accept-Language header associated with an IP subnet is a data point that can be used to protect against scraping. There are a lot more data points though and more aggressive fingerprinting can effectively destroy scraping.

hartleybrody 9y ago
The counterpoint (and what this article mentioned in several places) is that the "more aggressive fingerprinting" techniques can have high false positive rates, and then the legitimate users of your site are going to end up thinking it's broken or the data never loads because they're inadvertently triggering your fingerprinting. It feels pretty ridiculous to tell a user "you can't use our site because your system language settings don't match our IP list for that area." Sidestepping the fact that assuming you can tell someone's language from the geo mapping of their IP address is already pretty problematic.[0] [0]: https://medium.com/@kristopolous/stop-guessing-languages-based-on-ip-address-3862464b97c7 https://medium.com/@kristopolous/stop-guessing-languages-bas...
dsp1234 9y ago
If I can navigate to it using a normal Chrome instance, then a headless instance is going to have all of the same information (accept-language) as far as the server can detect. Chrome headless is Chrome and makes network calls in the exact same way. That's why client side headless detection even exists. So the only way in which what you suggest would work is if the system is so aggressive that it actually blocks a normal Chrome instance, which is more hostile that most systems are. But this allows a user to change settings until they do get a correct response back, and then just have the headless browser use those settings.
kbenson 9y ago
All the passive techniques are much harder to reasons about, but much easier to match. You just look at the complete request/response headers, make sure you match them, and have some good sources to request from. Much harder is stuff like Distil's script injection, where they transparently inject script tags that do fingerprinting, and they obfuscate the code that does so annoyingly (it's not really hard to reverse, just time consuming and and annoying). They pair this with being a bit more user friendly by redirecting you to a CAPTCHA page if your fingerprinting hits some threshold which if you answer redirects you to the page you wanted, so users experience and inconvenience if there's a false positive, but still get access to what they wanted. I was able to get around most of the passive fairly easily with Perl and LWP, and even the active stuff and CAPTCHA redirects (cookie_jar all serialized to DB so I could store request and represent it to a user to answer), but once they started tweaking their fingerprint script ever couple months/weeks that's when the equation shifted. Distil, as a solutions provider, gets to amortize their changes across all their customers, while I would have to spend the time de-obfuscating it. They could just assign a person to change it once a week and they would effectively halve my time to get any real work done, so without a collective effort of some sorts to combat them, I saw the writing on the wall. :/ The sad thing is that when we moved to API access, their APIs are hampered to the degree that it actually takes two orders of magnitude more requests each minute for a fraction of the accuracy (I was able to query changes over the last couple minutes previously, and now I have to query the entire item set of a subset of all containers, when there are tens of thousands of containers). :/ Lose lose, since our use case isn't even the main reason the site wanted to block scrapers.
- ThrustVectoring 9y ago
  Did you do a cost estimate for the "Wizard of Oz" solution of having real people with real browsers (and a script to pull data from the site?) Might have been worthwhile.
  - kbenson 9y ago
    It was actually two changes, one which required a much more intensive request regime because of a public caching (SOLR) system change, and then the much more aggressive scraping detection. The first caused us to change from requesting data 78 times an hour (items changed within 2 minutes requested every minute, 6 minutes of changes every 5 minutes, 11 minutes of changes every 10 minutes for overlapping coverage) to many thousands of checks an hour for much less accurate information. In the end, we ended up doing very targeted checks and much less accuracy for different classes of items. Having people actually do the checking just wouldn't be feasible for our size and resources (very small company, <10 employees), even through mechanical turk (I suspect).
sigotirandolas 9y ago
But certainly scapper blockers also have a big disadvantage, which is that the number of (reliable) scrapper detection techniques is finite, and probably pretty small. It's relatively possible (though no easy task) to find most of them by going through the Chromium source and looking at what is done differently when running in headless mode. And you can also probably find more by occasionally running some more real browser and doing differential execution of the JavaScript code. And once all trivial detection techniques can by bypassed, it becomes exponentially more difficult for scrapper blockers to find a new one.

hartleybrody 9y ago

The counterpoint (and what this article mentioned in several places) is that the "more aggressive fingerprinting" techniques can have high false positive rates, and then the legitimate users of your site are going to end up thinking it's broken or the data never loads because they're inadvertently triggering your fingerprinting. It feels pretty ridiculous to tell a user "you can't use our site because your system language settings don't match our IP list for that area." Sidestepping the fact that assuming you can tell someone's language from the geo mapping of their IP address is already pretty problematic.[0] [0]: https://medium.com/@kristopolous/stop-guessing-languages-based-on-ip-address-3862464b97c7 https://medium.com/@kristopolous/stop-guessing-languages-bas...

dsp1234 9y ago

If I can navigate to it using a normal Chrome instance, then a headless instance is going to have all of the same information (accept-language) as far as the server can detect. Chrome headless is Chrome and makes network calls in the exact same way. That's why client side headless detection even exists. So the only way in which what you suggest would work is if the system is so aggressive that it actually blocks a normal Chrome instance, which is more hostile that most systems are. But this allows a user to change settings until they do get a correct response back, and then just have the headless browser use those settings.

kbenson 9y ago

All the passive techniques are much harder to reasons about, but much easier to match. You just look at the complete request/response headers, make sure you match them, and have some good sources to request from. Much harder is stuff like Distil's script injection, where they transparently inject script tags that do fingerprinting, and they obfuscate the code that does so annoyingly (it's not really hard to reverse, just time consuming and and annoying). They pair this with being a bit more user friendly by redirecting you to a CAPTCHA page if your fingerprinting hits some threshold which if you answer redirects you to the page you wanted, so users experience and inconvenience if there's a false positive, but still get access to what they wanted. I was able to get around most of the passive fairly easily with Perl and LWP, and even the active stuff and CAPTCHA redirects (cookie_jar all serialized to DB so I could store request and represent it to a user to answer), but once they started tweaking their fingerprint script ever couple months/weeks that's when the equation shifted. Distil, as a solutions provider, gets to amortize their changes across all their customers, while I would have to spend the time de-obfuscating it. They could just assign a person to change it once a week and they would effectively halve my time to get any real work done, so without a collective effort of some sorts to combat them, I saw the writing on the wall. :/ The sad thing is that when we moved to API access, their APIs are hampered to the degree that it actually takes two orders of magnitude more requests each minute for a fraction of the accuracy (I was able to query changes over the last couple minutes previously, and now I have to query the entire item set of a subset of all containers, when there are tens of thousands of containers). :/ Lose lose, since our use case isn't even the main reason the site wanted to block scrapers.

ThrustVectoring 9y ago
Did you do a cost estimate for the "Wizard of Oz" solution of having real people with real browsers (and a script to pull data from the site?) Might have been worthwhile.
- kbenson 9y ago
  It was actually two changes, one which required a much more intensive request regime because of a public caching (SOLR) system change, and then the much more aggressive scraping detection. The first caused us to change from requesting data 78 times an hour (items changed within 2 minutes requested every minute, 6 minutes of changes every 5 minutes, 11 minutes of changes every 10 minutes for overlapping coverage) to many thousands of checks an hour for much less accurate information. In the end, we ended up doing very targeted checks and much less accuracy for different classes of items. Having people actually do the checking just wouldn't be feasible for our size and resources (very small company, <10 employees), even through mechanical turk (I suspect).

sigotirandolas 9y ago

But certainly scapper blockers also have a big disadvantage, which is that the number of (reliable) scrapper detection techniques is finite, and probably pretty small. It's relatively possible (though no easy task) to find most of them by going through the Chromium source and looking at what is done differently when running in headless mode. And you can also probably find more by occasionally running some more real browser and doing differential execution of the JavaScript code. And once all trivial detection techniques can by bypassed, it becomes exponentially more difficult for scrapper blockers to find a new one.