3 ms·

Does shot-scraper have a work around for sites that detect headless chrome? ie. news.com.au , nowsecure.nl

• simonw 3y ago
  No, nothing like that. I wonder how that detection works? I tried this and it took a shot of a "bot detected" screen: shot-scraper https://news.com.au/ But when I used interactive mode I could take the screenshot - run this: shot-scraper -i https://news.com.au/ It opens a Chrome window. Then hit "enter" in the CLI tool to take the screenshot.
  • simonw 3y ago
    Got this to work! shot-scraper https://news.com.au/ \ --init-script 'delete Object.getPrototypeOf(navigator).webdriver' \ --user-agent 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0' Code and screenshots and prototype --init-script feature here: https://github.com/simonw/shot-scraper/issues/147#issuecomment-2017196407 https://github.com/simonw/shot-scraper/issues/147#issuecomme...
    • nojs 3y ago
      If you want to go down the nowsecure.nl rabbithole (often used as a benchmark for passing bot detection) [1] is a good resource. It includes a few fixes that undetected-chromedriver doesn’t. 1. https://seleniumbase.io/help_docs/uc_mode/#uc-mode https://seleniumbase.io/help_docs/uc_mode/#uc-mode
    • bonestamp2 3y ago
      Nice! I was going to say that using experimental headless mode with the "User-Agent Switcher for Chrome" plugin might work too.
    • jaimex2 3y ago
      Love your work!