3 ms·
Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g.
by _delirium 16y ago
Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right.
I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
- fizx 16y agoSelenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
- il 16y agoSecond this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.