3 ms·
I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk
by ohthatsnotright 3y ago
I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients.
That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering).
[0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e...
[1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
- generalizations 3y agoWhat are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.
- horo 3y agoKasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
- ilyazub 3y agoReminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747 https://news.ycombinator.com/item?id=34285747