4 ms·
You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.
by robinduckett 16y ago
You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.
- _delirium 16y agoSounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right. I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
- fizx 16y agoSelenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
- il 16y agoSecond this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.
- il 16y agoIs that due to a technical limitation or a security issue? Would everything need to run in a separate sandbox if the JS was from a trusted source(i.e. google)?
- deleted 16y ago[deleted]