3 ms·

UserAgent detection is in the "old" groups, is specific to having your userAgent be "HeadlessChrome", and is no longer recommended. The new triggers are 'naviga

UserAgent detection is in the "old" groups, is specific to having your userAgent be "HeadlessChrome", and is no longer recommended. The new triggers are 'navigator.webdriver', or a chrome extension specific object, or specific permissions being set, none of which are relevant to or impacted by any the scenarios you are highlighting... Of course the JS itself can fail due to incongruent browser behaviour... but why would you trigger a bot obfuscation routine based on a failed JS call? That is the gross incompetence I was referring to, and it's hard to call basic errors a lack of basic testing anything but. Downvotes aside, the kinds of f-ups you're speculating about here are at the level of knowing how true/false works in JS. And, no, there really are not valid reasons for users to be adding specific properties on their navigation objects to flag for headless, or use specific extension objects that report the use of headless automation, if they aren't. There is no valid reason you should set your Edge userAgent to "HeadlessChrome", either. That's not an angry lost user, friend, that is an upset unauthorized third-party content scraper. I work with Open Data, so I don't care, but some sites for-realsies do.

sovok_x 9y ago
Providing users with fake data is never a good idea because it can be, and probably will be, used against you in the long run. Plus no sane evil scrapper uses default referrer and no masking so misfires are realistically possible within a thin line needed to detect them. In any case, users can do whatever they want with their client and expect the service to work properly. If you detect abuse you should block or captcha them but the fact of them being a possible bot doesn't really call for such drastic measure. It's the second worst approach after serving hindering scripts to them. Disclaimer: I haven't downvoted you as I don't downvote things prompting a discussion.

sovok_x 9y ago

Providing users with fake data is never a good idea because it can be, and probably will be, used against you in the long run. Plus no sane evil scrapper uses default referrer and no masking so misfires are realistically possible within a thin line needed to detect them. In any case, users can do whatever they want with their client and expect the service to work properly. If you detect abuse you should block or captcha them but the fact of them being a possible bot doesn't really call for such drastic measure. It's the second worst approach after serving hindering scripts to them. Disclaimer: I haven't downvoted you as I don't downvote things prompting a discussion.