3 ms·

Is there a reason why the crawling and browser automation people don't just patch the browser to be controlled with no possibility of detection? The web page i

Is there a reason why the crawling and browser automation people don't just patch the browser to be controlled with no possibility of detection? The web page is heavily restricted in what is can access through various interfaces and you can feed it anything you want by patching the browser. Once you do that the problem becomes just simulating a legitimate user to a sufficient degree. I wonder if that's what's already happening with CDP and ReCAPTCHA and hCaptcha - the two services mentioned that are strong and a problem. Are they detecting the "Stealth" or is it just the lack of user activity and reputation? Is CDP by itself detectable by some means?

seleniumbase 2y ago
Patching chromedriver is a lot easier than patching the browser. Plus, if you're just using a regular Chrome browser for the automation, then there's nothing to patch. Automated CDP calls aren't detectable if they don't leave any trace of automation activity. However, since Google created CDP, they might have ways of detecting automated CDP in ways that other services cannot.
- coppsilgold 2y ago
  What about faking mouse movement from inside the browser? PyAutoGUI is not the right way to be doing this for interacting with JavaScript that has no hope of interrogating user operating system GUI interactions. And it seems like it would be important to try and adopt user-like mouse movement since JavaScript has access to this information.
  - mintzworld 2y ago
    PyAutoGUI is the optimal tool for clicking things inside of closed shadow-root elements, which are hidden to JavaScript. Can use CDP for clicking other elements.
ghxst 2y ago
The reason in my experience is that there's a high barrier of entry for most devs when it comes to setting up an environment for Chromium and a workflow for patches that still allows you to quickly and easily pull in and apply upstream changes whenever a new Chromium version releases. In reality, if you know how to use CDP correctly and you have control over the environment that you run the browser in, you have to make very few browser patches. What I mean with using CDP correctly is that, yes it is detectable to a certain extent but it comes down to things like enabling Runtime domain for example which you can easily mitigate in your own solution but is something that libraries like puppeteer / playwright often do out of the box (this is where the "stealth" versions of these libraries come in, they will either mitigate by disabling features or use some hacky approaches to instrument the JS that runs on the pages). Then when you move into an environment that is a lot more stripped down (let's say from your home machine to docker) now you run into A LOT of issues that you definitely are better off fixing with browser patches, however figuring out what those issues are and how to fix them is a huge feat in itself and often will require you to have the ability to reverse engineer things like Cloudflare, Akamai and other anti bot vendors just to know what leaks you still have to patch. It doesn't help that there is no end to misinformed articles on things like "browser fingerprinting" that you encounter when you try to solve your issues the first time you encounter them, a lot of articles based on nothing but superstition, articles that basically say "proxies are never good enough", "captchas are getting out of hand" that get things wrong and will just eat away at your sanity while trying to debug issues. This is long enough of a rant already but maybe offers you some insight, if you have any specific questions feel free to ask.
- seleniumbase 2y ago
  The biggest issue with going from a home machine to a server is that you may lose having a "residential IP address", which is something that you'll want to have in order to prevent automation from being blocked outright. Hence the popularity of residential proxies. However, some servers live in a residential IP space, which makes them optimal for running web automation in. As was partially covered in https://www.youtube.com/watch?v=Mr90iQmNsKM https://www.youtube.com/watch?v=Mr90iQmNsKM, GitHub Actions appears to live in a "Residential IP space", which makes it a good server choice for web automation.

seleniumbase 2y ago

Patching chromedriver is a lot easier than patching the browser. Plus, if you're just using a regular Chrome browser for the automation, then there's nothing to patch. Automated CDP calls aren't detectable if they don't leave any trace of automation activity. However, since Google created CDP, they might have ways of detecting automated CDP in ways that other services cannot.

coppsilgold 2y ago
What about faking mouse movement from inside the browser? PyAutoGUI is not the right way to be doing this for interacting with JavaScript that has no hope of interrogating user operating system GUI interactions. And it seems like it would be important to try and adopt user-like mouse movement since JavaScript has access to this information.
- mintzworld 2y ago
  PyAutoGUI is the optimal tool for clicking things inside of closed shadow-root elements, which are hidden to JavaScript. Can use CDP for clicking other elements.

ghxst 2y ago

The reason in my experience is that there's a high barrier of entry for most devs when it comes to setting up an environment for Chromium and a workflow for patches that still allows you to quickly and easily pull in and apply upstream changes whenever a new Chromium version releases. In reality, if you know how to use CDP correctly and you have control over the environment that you run the browser in, you have to make very few browser patches. What I mean with using CDP correctly is that, yes it is detectable to a certain extent but it comes down to things like enabling Runtime domain for example which you can easily mitigate in your own solution but is something that libraries like puppeteer / playwright often do out of the box (this is where the "stealth" versions of these libraries come in, they will either mitigate by disabling features or use some hacky approaches to instrument the JS that runs on the pages). Then when you move into an environment that is a lot more stripped down (let's say from your home machine to docker) now you run into A LOT of issues that you definitely are better off fixing with browser patches, however figuring out what those issues are and how to fix them is a huge feat in itself and often will require you to have the ability to reverse engineer things like Cloudflare, Akamai and other anti bot vendors just to know what leaks you still have to patch. It doesn't help that there is no end to misinformed articles on things like "browser fingerprinting" that you encounter when you try to solve your issues the first time you encounter them, a lot of articles based on nothing but superstition, articles that basically say "proxies are never good enough", "captchas are getting out of hand" that get things wrong and will just eat away at your sanity while trying to debug issues. This is long enough of a rant already but maybe offers you some insight, if you have any specific questions feel free to ask.

seleniumbase 2y ago
The biggest issue with going from a home machine to a server is that you may lose having a "residential IP address", which is something that you'll want to have in order to prevent automation from being blocked outright. Hence the popularity of residential proxies. However, some servers live in a residential IP space, which makes them optimal for running web automation in. As was partially covered in https://www.youtube.com/watch?v=Mr90iQmNsKM https://www.youtube.com/watch?v=Mr90iQmNsKM, GitHub Actions appears to live in a "Residential IP space", which makes it a good server choice for web automation.