4 ms·
Got this to work! shot-scraper https://news.com.au/ \ --init-script 'delete Object.getPrototypeOf(navigator).webdriver' \ --user-agent 'Mozilla
by simonw 3y ago
Got this to work!
shot-scraper https://news.com.au/ \
--init-script 'delete Object.getPrototypeOf(navigator).webdriver' \
--user-agent 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0'
Code and screenshots and prototype --init-script feature here: https://github.com/simonw/shot-scraper/issues/147#issuecomment-2017196407 https://github.com/simonw/shot-scraper/issues/147#issuecomme...
- nojs 3y agoIf you want to go down the nowsecure.nl rabbithole (often used as a benchmark for passing bot detection) [1] is a good resource. It includes a few fixes that undetected-chromedriver doesn’t. 1. https://seleniumbase.io/help_docs/uc_mode/#uc-mode https://seleniumbase.io/help_docs/uc_mode/#uc-mode
- bonestamp2 3y agoNice! I was going to say that using experimental headless mode with the "User-Agent Switcher for Chrome" plugin might work too.
- jaimex2 3y agoLove your work!