5 ms·

How useful are scrapers that don't execute Javascript these days? I find Selenium + PhantomJS (now Chrome Headless I guess) is pretty easy to drive from Python,

by vosper 9y ago

How useful are scrapers that don't execute Javascript these days? I find Selenium + PhantomJS (now Chrome Headless I guess) is pretty easy to drive from Python, and it works everywhere because it's a real browser.

dhruvkar 9y ago
Phantom still gets blocked, as it reveals itself in the header. I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Selenium, backed by Postgres. It's as close you can get to scripting a real browser.
- rectangletangle 9y ago
  You can set the user-agent with Phantom var webPage = require('webpage'); var page = webPage.create(); page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36'; http://phantomjs.org/api/webpage/property/settings.html http://phantomjs.org/api/webpage/property/settings.html
  - dhruvkar 9y ago
    While that's true, user-agent isn't the only thing in the header that reveals PhantomJS[0]. You could take the time to build in spoofs for these issues. But for testing (and scraping), you're going to be better off if your headless browser is the same as your GUI browser. 0: https://blog.shapesecurity.com/2015/01/22/detecting-phantomjs-based-visitors/ https://blog.shapesecurity.com/2015/01/22/detecting-phantomj...
- eriknstr 9y ago
  I think I read recently that Chrome / Chromium is now able to run without having to use xvfb, so now truly headless.
iogf 9y ago
These ones are slower in most cases, it seems for some situations the ones that dont execute js would better do the job.
rectangletangle 9y ago
They're still surprisingly useful, however it depends a lot on your use case. In my case I've scraped quite a bit from Wikipedia (not everything is available in a clean API) and other sources this way.
- danpalmer 9y ago
  Instead of scraping Wikipedia have you had a look at http://wiki.dbpedia.org/ http://wiki.dbpedia.org/ ? They provide a SPARQL endpoint for querying the knowledge graph on Wikipedia.
  - ConfucianNardin 9y ago
    Depending on your use case you can also download full database dumps (https://dumps.wikimedia.org/ https://dumps.wikimedia.org/).
Mic92 9y ago
In my experience when the page is rendered with javascript, there is often a json "API", which is even easier to use. Web browsers are often too slow and load content I am not interested in.
- penetrarthur 9y ago
  I've done some scraping recently and 99% of sites don't have the json api
danpalmer 9y ago
We scrape a significant amount of highly structured data from a large number of websites (in our case, inventory from ecommerce sites), and have yet to find a site where we needed to use a headless browser. So far we've managed with just lxml. That also includes driving some checkout processes as well (although we don't do this for everyone). We used to use Selenium + Firefox for the checkout processes (run manually), but it was too much maintenance overhead so we switched to requests+lxml. We generally find the more "single page app" a website is, the easier it is to scrape, because we can just use the API that's backing the SPA directly, rather than parsing data out of the HTML.

by vosper 9y ago

dhruvkar 9y ago

Phantom still gets blocked, as it reveals itself in the header. I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Selenium, backed by Postgres. It's as close you can get to scripting a real browser.

rectangletangle 9y ago
You can set the user-agent with Phantom var webPage = require('webpage'); var page = webPage.create(); page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36'; http://phantomjs.org/api/webpage/property/settings.html http://phantomjs.org/api/webpage/property/settings.html
- dhruvkar 9y ago
  While that's true, user-agent isn't the only thing in the header that reveals PhantomJS[0]. You could take the time to build in spoofs for these issues. But for testing (and scraping), you're going to be better off if your headless browser is the same as your GUI browser. 0: https://blog.shapesecurity.com/2015/01/22/detecting-phantomjs-based-visitors/ https://blog.shapesecurity.com/2015/01/22/detecting-phantomj...
eriknstr 9y ago
I think I read recently that Chrome / Chromium is now able to run without having to use xvfb, so now truly headless.

iogf 9y ago

These ones are slower in most cases, it seems for some situations the ones that dont execute js would better do the job.

rectangletangle 9y ago

They're still surprisingly useful, however it depends a lot on your use case. In my case I've scraped quite a bit from Wikipedia (not everything is available in a clean API) and other sources this way.

danpalmer 9y ago
Instead of scraping Wikipedia have you had a look at http://wiki.dbpedia.org/ http://wiki.dbpedia.org/ ? They provide a SPARQL endpoint for querying the knowledge graph on Wikipedia.
- ConfucianNardin 9y ago
  Depending on your use case you can also download full database dumps (https://dumps.wikimedia.org/ https://dumps.wikimedia.org/).

Mic92 9y ago

In my experience when the page is rendered with javascript, there is often a json "API", which is even easier to use. Web browsers are often too slow and load content I am not interested in.

penetrarthur 9y ago
I've done some scraping recently and 99% of sites don't have the json api

danpalmer 9y ago

We scrape a significant amount of highly structured data from a large number of websites (in our case, inventory from ecommerce sites), and have yet to find a site where we needed to use a headless browser. So far we've managed with just lxml. That also includes driving some checkout processes as well (although we don't do this for everyone). We used to use Selenium + Firefox for the checkout processes (run manually), but it was too much maintenance overhead so we switched to requests+lxml. We generally find the more "single page app" a website is, the easier it is to scrape, because we can just use the API that's backing the SPA directly, rather than parsing data out of the HTML.