4 ms·

I have a question: Does scraping like this execute Javascript on the scraped page? Am I able to access the output of Javascript/AJAX on that page? As far as I

by il 16y ago

I have a question: Does scraping like this execute Javascript on the scraped page? Am I able to access the output of Javascript/AJAX on that page? As far as I know this is impossible with any other server-side scraping technology. If so, that would be amazingly useful for a couple of my side projects, much easier than parsing their Javascript code and extracting the info I need.

robinduckett 16y ago
You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.
- _delirium 16y ago
  Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right. I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
  - fizx 16y ago
    Selenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
  - il 16y ago
    Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.
- il 16y ago
  Is that due to a technical limitation or a security issue? Would everything need to run in a separate sandbox if the JS was from a trusted source(i.e. google)?
- deleted 16y ago
  [deleted]
vially 16y ago
If you are using Java you could use HtmlUnit which has fairly good JavaScript support (including AJAX).

by il 16y ago

robinduckett 16y ago

You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.

_delirium 16y ago
Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right. I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
- fizx 16y ago
  Selenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
- il 16y ago
  Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.
il 16y ago
Is that due to a technical limitation or a security issue? Would everything need to run in a separate sandbox if the JS was from a trusted source(i.e. google)?
deleted 16y ago
[deleted]

vially 16y ago

If you are using Java you could use HtmlUnit which has fairly good JavaScript support (including AJAX).