3 ms·

someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]

by nhggfu 3y ago

ohthatsnotright 3y ago
You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode. edit: Firefox Nightly because it gives you much deeper access to Firefox internals.
- nhggfu 3y ago
  interesting. would love to see some examples / code
  - ohthatsnotright 3y ago
    I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
    - generalizations 3y ago
      What are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.
      - horo 3y ago
        Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
        ilyazub 3y ago
        Reminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747 https://news.ycombinator.com/item?id=34285747
michaelt 3y ago
There's actually an (admittedly small) data scraping industry. Let's say you are Wal-Mart, and you'd like to know which of the products you sell are available cheaper at Target. Or which neighbourhoods they deliver to that you don't, or which stores have longer opening hours than yours, or whatever. You can't legally exchange data with Target directly, that would risk making you an illegal price-fixing cartel. You can legally visit their website, but you don't feel like matching up 10,000+ products. Instead, you call up a business intelligence firm who has already scraped your site and theirs and matched the products up. They'll send you a CSV, for a price.

someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]

by nhggfu 3y ago

someone got tweak we can use for puppeteer to achieve this effect? [kinda LOL @ asking people who SCRAPE DATA for free to pay for an API, no?]

ohthatsnotright 3y ago

You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode. edit: Firefox Nightly because it gives you much deeper access to Firefox internals.

nhggfu 3y ago
interesting. would love to see some examples / code
- ohthatsnotright 3y ago
  I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients. That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering). [0] https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html https://firefox-source-docs.mozilla.org/overview/gecko.html
  - generalizations 3y ago
    What are some sites that have good bot detection? I'd be interested in just seeing how my scraper holds up against them.
    - horo 3y ago
      Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
      - ilyazub 3y ago
        Reminded me the the Kasada VM reverse-engineering article: https://news.ycombinator.com/item?id=34285747 https://news.ycombinator.com/item?id=34285747

michaelt 3y ago

There's actually an (admittedly small) data scraping industry. Let's say you are Wal-Mart, and you'd like to know which of the products you sell are available cheaper at Target. Or which neighbourhoods they deliver to that you don't, or which stores have longer opening hours than yours, or whatever. You can't legally exchange data with Target directly, that would risk making you an illegal price-fixing cartel. You can legally visit their website, but you don't feel like matching up 10,000+ products. Instead, you call up a business intelligence firm who has already scraped your site and theirs and matched the products up. They'll send you a CSV, for a price.