4 ms·
Correct me if I’m wrong but neither one supports Javascript rendered pages? You’re right in the overhead though; I’d stay miles away from Electron for scraping
by dstick 8y ago
Correct me if I’m wrong but neither one supports Javascript rendered pages?
You’re right in the overhead though; I’d stay miles away from Electron for scraping but you’ll need more than a CURL wrapper to properly fetch data in all shapes and sizes :) Headless Chromium does do the trick in that regard.
- danpalmer 8y agoWith web scraping you typically don’t want the visuals anyway. JS rendered applications are usually easier to scrape because they have data in a more raw or canonical format available somewhere to do that rendering.
- karmelapple 8y agoPlenty of websites will only render the content fully after some JavaScript runs, so to properly scrape them you do indeed need a browser to process the JS. This includes text content.
- adamzochowski 8y agoJavascript rendered pages load JS which then in turn calls some rest API to get data and use that to render contents. Web scraper stops scraping the html, but calls and scrapes the rest api endpoint.