4 ms·

The code posted on the site fails simply on incongruity of the JS behaviour (window.chrome) and userAgent. I can see how this can fail with common user setting

The code posted on the site fails simply on incongruity of the JS behaviour (window.chrome) and userAgent. I can see how this can fail with common user setting userAgent to Chrome on Edge or Firefox for some compatibility reason or just forgetting to turn off old referrer override. There can be other valid reasons it will fail if user is not bot that I miss. And BLAM they'll get all wrong data for no reason... You may call it gross incompetence or whatever but this method will get you one angry lost user at the time.

bonesss 9y ago
UserAgent detection is in the "old" groups, is specific to having your userAgent be "HeadlessChrome", and is no longer recommended. The new triggers are 'navigator.webdriver', or a chrome extension specific object, or specific permissions being set, none of which are relevant to or impacted by any the scenarios you are highlighting... Of course the JS itself can fail due to incongruent browser behaviour... but why would you trigger a bot obfuscation routine based on a failed JS call? That is the gross incompetence I was referring to, and it's hard to call basic errors a lack of basic testing anything but. Downvotes aside, the kinds of f-ups you're speculating about here are at the level of knowing how true/false works in JS. And, no, there really are not valid reasons for users to be adding specific properties on their navigation objects to flag for headless, or use specific extension objects that report the use of headless automation, if they aren't. There is no valid reason you should set your Edge userAgent to "HeadlessChrome", either. That's not an angry lost user, friend, that is an upset unauthorized third-party content scraper. I work with Open Data, so I don't care, but some sites for-realsies do.
- sovok_x 9y ago
  Providing users with fake data is never a good idea because it can be, and probably will be, used against you in the long run. Plus no sane evil scrapper uses default referrer and no masking so misfires are realistically possible within a thin line needed to detect them. In any case, users can do whatever they want with their client and expect the service to work properly. If you detect abuse you should block or captcha them but the fact of them being a possible bot doesn't really call for such drastic measure. It's the second worst approach after serving hindering scripts to them. Disclaimer: I haven't downvoted you as I don't downvote things prompting a discussion.

bonesss 9y ago

UserAgent detection is in the "old" groups, is specific to having your userAgent be "HeadlessChrome", and is no longer recommended. The new triggers are 'navigator.webdriver', or a chrome extension specific object, or specific permissions being set, none of which are relevant to or impacted by any the scenarios you are highlighting... Of course the JS itself can fail due to incongruent browser behaviour... but why would you trigger a bot obfuscation routine based on a failed JS call? That is the gross incompetence I was referring to, and it's hard to call basic errors a lack of basic testing anything but. Downvotes aside, the kinds of f-ups you're speculating about here are at the level of knowing how true/false works in JS. And, no, there really are not valid reasons for users to be adding specific properties on their navigation objects to flag for headless, or use specific extension objects that report the use of headless automation, if they aren't. There is no valid reason you should set your Edge userAgent to "HeadlessChrome", either. That's not an angry lost user, friend, that is an upset unauthorized third-party content scraper. I work with Open Data, so I don't care, but some sites for-realsies do.

sovok_x 9y ago
Providing users with fake data is never a good idea because it can be, and probably will be, used against you in the long run. Plus no sane evil scrapper uses default referrer and no masking so misfires are realistically possible within a thin line needed to detect them. In any case, users can do whatever they want with their client and expect the service to work properly. If you detect abuse you should block or captcha them but the fact of them being a possible bot doesn't really call for such drastic measure. It's the second worst approach after serving hindering scripts to them. Disclaimer: I haven't downvoted you as I don't downvote things prompting a discussion.