3 ms·
Selenium lets you add random delays between your actions which could help avoid triggering a firewall to block you. Good practice anyway so you don't overload
by dwd 6y ago
Selenium lets you add random delays between your actions which could help avoid triggering a firewall to block you.
Good practice anyway so you don't overload the site and find your logs empty or full of gaps.
- bdcravens 6y agoGood approach, but advanced Selenium detection goes beyond heuristics. Selenium injects JavaScript into the page to function, and the presence of this is how Selenium is detected.
- dwd 6y agoInteresting, I've worked on both sides of scraping and protecting content but hadn't really considered checking for JavaScript frameworks as a trigger. I'm assuming this is something you could configure in a F5 that also injects its own JavaScript? Randomising field names, seeding hidden bogus data and messing with element order was more what I would look at once a persistent scraper was using enough IPs to get around rate limits.
- bdcravens 6y agoI believe you could recompile Selenium with different names to get around it, but since Puppeteer uses CDP baked into the browser, no injection is necessary, bypassing alot of this.