Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers

Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342 https://news.ycombinator.com/item?id=25379342