3 ms·
Nice. It would be pretty cool to have an "run javascript and wait for network idle" optional for scraping js-requiring websites. Can selenium do this? Headless
by ivan_ah 9y ago
Nice. It would be pretty cool to have an "run javascript and wait for network idle" optional for scraping js-requiring websites. Can selenium do this? Headless chrome?
I personally use bs4 for web scraping and it works pretty well, but if there was an option to also do js with a sane API, I'd switch in a heartbeat.
- bshimmin 9y agoPuppeteer has "networkIdle0" (0 network connections for 500ms) and "networkIdle2" (no more than 2 network connections for 500ms). My experiences with it have been very positive.
- _bohm 9y agoSelenium can absolutely be used for this, the caveat being that it is much slower than using a regular HTML parser. In my experience, it's best to milk as much as you possibly can out of a site's plain HTML and APIs, only resorting to Selenium where it's absolutely necessary.
- nojvek 9y agoWe use puppeteer for our smoke tests. Ensure all network requests load, no JavaScript errors, take screenshot of page and simple validations so we know our deploys aren’t borked. I really love puppeteer over selenium. Much deeper control.