3 ms·

You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistin

You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode. edit: Firefox Nightly because it gives you much deeper access to Firefox internals.

nhggfu 3y ago
interesting. would love to see some examples / code
- ohthatsnotright 3y ago
  I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
  - generalizations 3y ago
    What are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.
    - horo 3y ago
      Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
      - ilyazub 3y ago
        Reminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747 https://news.ycombinator.com/item?id=34285747

nhggfu 3y ago

interesting. would love to see some examples / code

ohthatsnotright 3y ago
I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
- generalizations 3y ago
  What are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.
  - horo 3y ago
    Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
    - ilyazub 3y ago
      Reminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747 https://news.ycombinator.com/item?id=34285747