3 ms·
How does one do scraping properly on dynamic client side rendered pages?
by max_ 6y ago
How does one do scraping properly on dynamic client side rendered pages?
- jmt_ 6y agoUsually the approach is to use a headless browser. The headless browser instance runs purely in memory without a GUI then renders the website you're interested in. Then, it comes down to regular DOM parsing. A common library that I enjoy is Selenium with Python.
- Topgamer7 6y agoMost SPA pages will honour direct uri requests and route you properly in javascript. You just need to have your scraping pipeline use phantomJS or selenium to wait until the page stops loading then scrape the html. Although it might just be easier to scrape their api endpoints directly instead of mucking with html if its a dynamic page. The data is structured that way, and easier to query.