4 ms·
PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere,
by tyingq 6y ago
PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API.
As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days.
https://github.com/pyppeteer/pyppeteer https://github.com/pyppeteer/pyppeteer
- daolf 6y agoI think you'd be surprised by the amount of website you can scrape without an headless browser. Even Google SERP can be scraped with a simple HTTP client.
- tyingq 6y agoThat's what I meant by narrow...a known set of sites and data you want to extract. I imagine, for example, building on the SERP example might hit a wall if you added logged in vs not logged SERPS, iterating over carousel data, reading advertisement data etc.
- daolf 6y agolog in wall can easily be bypassed with an HTTP client by setting the correct auth header. From what I can observe, 2/3 websites can be scraped without using a headless browser.
- hakube 6y agoHow do you deal with JavaScript then?
- JackC 6y agoThere's an official Python library for Playwright as well: https://github.com/microsoft/playwright-python https://github.com/microsoft/playwright-python
- pknerd 6y agoI am often contacted by people who ask me to scrape a dynamic/JS rendered websites. You might be surprised to know that many of such dynamic websites are actually depending on some API end-point which is being accessed via some AJAX like functionality which you can access directly and get the required data. I often faced the situation where data was not fetched via some external source was already available either in data-field or some JSON like structure hence no need to use Selenium with headless browser.
- tyingq 6y agoSure. This one happens not be Selenium.