8 ms·

Using jQuery and node.js to scrape html pages in 5 lines

133 points by Ainab 16y ago

forsaken 16y ago
Site appears down. Is node popular enough yet for the "Node doesn't scale" talk? :)
- drats 16y ago
  Yes, as cliched as it is, I think it's time. I couldn't use at least 6/10 of the node challenge top 10 when it hit the HN front page (and the rest were beset by bugs and didn't work - the pixel one where you form characters stopped showing the shape I was supposed to be trying to get into after a few rounds, and the robot war one never let me buy or release my wave of robots on Chrome or Firefox). Overall it was totally disappointing experience.
  - mnutt 16y ago
    I think there are a couple of factors here: 1. It was a 48-hour coding competition, so some bugs are to be expected. 2. The hosting for the apps in this competition was turned on literally the day before: http://twitter.com/joyent/status/22204412477 http://twitter.com/joyent/status/22204412477 I don't think that "node can't scale" will get much press, just because node is pretty performant and many people conflate performance and scalability. Node's asynchronous api would seem pretty good in a services-oriented architecture, and scalability is mostly about how the app is architected. (and in 48 hours, that probably means "quickly") On the other hand, node.js is still a very young platform and there are a lot of unsolved issues. As of right now there are only 3 or 4 dedicated node.js hosts that I know of, and all are in private beta. People are still trying to figure out best practices for design and testing. The thing that I took away from Node Knockout was that here are a bunch of really cool web app ideas that, given time, could be built with node. I think the apps were 'frozen' pending judging, but I know many of the top app teams were planning on improving them as soon as they had the chance.
- robinduckett 16y ago
  Not down for me :)
drats 16y ago
Strange popup: "Hello, i see you are coming from hacker news. the article you clicked on was most certainly not submitted by nodejitsu. news.ycombinator has a long history of squashing articles and submitters that aren't funded by y-comb. most of this is done through their "silent" banning and censoring mechanisms, that leave people not even realizing they have been silenced. i hope you enjoy this article, and remember that HN is extremely biased and that you should keep your horizons broad." While I would agree that HN is bias towards YC-funded projects I would not agree that it is biased against non-YC projects or news. In fact, the majority of the items on HN are non-YC. This also follows for submitters and commenters for the year or more I've been here. On a different note. Hpricot is not representative of Ruby scraping anymore - nokogiri (http://nokogiri.org/ http://nokogiri.org/) is where it's at. Which has a Hpricot translation layer if you need to change. Even when I decided to solidify on Python for everything else I will still go back to Ruby just for nokogiri when it comes to scraping.
- steilpass 16y ago
  Are there any information/discussions about the "censoring mechanisms"?
  - iamelgringo 16y ago
    Over the years, I've occasionally heard of people getting kickbanned from HN. It doesn't happen that often, but when it does, it's been because of egregious trolling, spamming, etc... I've also heard of one or two voting rings that got booted as well. It doesn't happen too often, however. The community here usually has pretty high commenting standards, and there's peer pressure that encourages good behavior.
    - thaumaturgy 16y ago
      There are some individuals that are either around for a while, or just getting started, and for whatever arbitrary reason one day all of their comments and submissions will be auto-dead. They of course aren't notified at all, so they're quite happy to keep adding comments to the site, all of which stay at 1 point and never get replied to (or even read by anyone except those with showdead on). Since there's no messaging system on this site, if they haven't bothered to add any contact information to their userpage then there's no way to let them know. I occasionally see these folks posting some pretty thoughtful comments while auto-dead, too. However, this has been a subject of discussion here a few times, and nothing has changed; I don't expect it to. I do expect that this site will be banned soon too.
fmw 16y ago
The article lists BeautifulSoup as the Python choice for scraping, but that isn't necessarily true. I'm using http://scrapy.org/ http://scrapy.org/, for example, which is a scraping framework that uses lxml and xpath by default.
- bmelton 16y ago
  I don't know if it was edited after your post, but it lists Scrapy right next to Beautiful Soup.
il 16y ago
I have a question: Does scraping like this execute Javascript on the scraped page? Am I able to access the output of Javascript/AJAX on that page? As far as I know this is impossible with any other server-side scraping technology. If so, that would be amazingly useful for a couple of my side projects, much easier than parsing their Javascript code and extracting the info I need.
- robinduckett 16y ago
  You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.
  - _delirium 16y ago
    Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right. I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
    - fizx 16y ago
      Selenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
    - il 16y ago
      Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.
robinduckett 16y ago
Hey guys. The Nodejitsu team and Marak (http://www.github.com/Marak http://www.github.com/Marak), the guy behind Nodejitsu are perma-banned from HN and can't respond to your queries. He sends his regards, and if you'd like to contact him visit the #Node.js IRC channel @ Freenode
fizx 16y ago
This reminds me, I ported the core ideas of the parsley scraping language to jQuery. http://github.com/fizx/pquery#readme http://github.com/fizx/pquery#readme
deleted 16y ago
[deleted]
- wmil 16y ago
  HN deletes posts to try to keeps things on topic. It's heavy handed, but there aren't a lot of ways to stop flame wars and pun threads.
  - pygy_ 16y ago
    I apologize. I deleted my former post, because the censorship accusations were based on an error on my side, I had been reading another comment thread related to the author of the post[1] and mixed things up. [1] http://news.ycombinator.com/item?id=1448309 http://news.ycombinator.com/item?id=1448309
knowtheory 16y ago
The article reads "The challenge with using these libraries is that they all have their own quirks that can make working with HTML, CSS and Javascript challenging." And that's true only if you only want to do page manipulation in Javascript. I'm perfectly happy with my page manipulation in Ruby w/ Nokogiri. Here's an example: (code formatting on HN sucks, so it's on my blog, apologies) http://blog.knowtheory.net/post/1074676060/xml-manipulation-in-6-lines-of-ruby http://blog.knowtheory.net/post/1074676060/xml-manipulation-...
deleted 16y ago
[deleted]
tcarnell 16y ago
Interesting, when I built http://cQuery.com http://cQuery.com (Content Query Engine), I investigated a number of options html parsing and content extraction options. I had played with Rhino and John Resigs env.js (http://ejohn.org/blog/bringing-the-browser-to-the-server/ http://ejohn.org/blog/bringing-the-browser-to-the-server/) to run jQuery server-side. For portability, performance and flexability I finally settled for writing my own HTML parser and CSS selection engine from scratch.
jfager 16y ago
Ignoring the drama: my current favorite scraping combo is NekoHtml underneath Scala's completely kickass combo of pattern matching and XML literals.

133 points by Ainab 16y ago

forsaken 16y ago

Site appears down. Is node popular enough yet for the "Node doesn't scale" talk? :)

drats 16y ago
Yes, as cliched as it is, I think it's time. I couldn't use at least 6/10 of the node challenge top 10 when it hit the HN front page (and the rest were beset by bugs and didn't work - the pixel one where you form characters stopped showing the shape I was supposed to be trying to get into after a few rounds, and the robot war one never let me buy or release my wave of robots on Chrome or Firefox). Overall it was totally disappointing experience.
- mnutt 16y ago
  I think there are a couple of factors here: 1. It was a 48-hour coding competition, so some bugs are to be expected. 2. The hosting for the apps in this competition was turned on literally the day before: http://twitter.com/joyent/status/22204412477 http://twitter.com/joyent/status/22204412477 I don't think that "node can't scale" will get much press, just because node is pretty performant and many people conflate performance and scalability. Node's asynchronous api would seem pretty good in a services-oriented architecture, and scalability is mostly about how the app is architected. (and in 48 hours, that probably means "quickly") On the other hand, node.js is still a very young platform and there are a lot of unsolved issues. As of right now there are only 3 or 4 dedicated node.js hosts that I know of, and all are in private beta. People are still trying to figure out best practices for design and testing. The thing that I took away from Node Knockout was that here are a bunch of really cool web app ideas that, given time, could be built with node. I think the apps were 'frozen' pending judging, but I know many of the top app teams were planning on improving them as soon as they had the chance.
robinduckett 16y ago
Not down for me :)

drats 16y ago

Strange popup: "Hello, i see you are coming from hacker news. the article you clicked on was most certainly not submitted by nodejitsu. news.ycombinator has a long history of squashing articles and submitters that aren't funded by y-comb. most of this is done through their "silent" banning and censoring mechanisms, that leave people not even realizing they have been silenced. i hope you enjoy this article, and remember that HN is extremely biased and that you should keep your horizons broad." While I would agree that HN is bias towards YC-funded projects I would not agree that it is biased against non-YC projects or news. In fact, the majority of the items on HN are non-YC. This also follows for submitters and commenters for the year or more I've been here. On a different note. Hpricot is not representative of Ruby scraping anymore - nokogiri (http://nokogiri.org/ http://nokogiri.org/) is where it's at. Which has a Hpricot translation layer if you need to change. Even when I decided to solidify on Python for everything else I will still go back to Ruby just for nokogiri when it comes to scraping.

steilpass 16y ago
Are there any information/discussions about the "censoring mechanisms"?
- iamelgringo 16y ago
  Over the years, I've occasionally heard of people getting kickbanned from HN. It doesn't happen that often, but when it does, it's been because of egregious trolling, spamming, etc... I've also heard of one or two voting rings that got booted as well. It doesn't happen too often, however. The community here usually has pretty high commenting standards, and there's peer pressure that encourages good behavior.
  - thaumaturgy 16y ago
    There are some individuals that are either around for a while, or just getting started, and for whatever arbitrary reason one day all of their comments and submissions will be auto-dead. They of course aren't notified at all, so they're quite happy to keep adding comments to the site, all of which stay at 1 point and never get replied to (or even read by anyone except those with showdead on). Since there's no messaging system on this site, if they haven't bothered to add any contact information to their userpage then there's no way to let them know. I occasionally see these folks posting some pretty thoughtful comments while auto-dead, too. However, this has been a subject of discussion here a few times, and nothing has changed; I don't expect it to. I do expect that this site will be banned soon too.

fmw 16y ago

The article lists BeautifulSoup as the Python choice for scraping, but that isn't necessarily true. I'm using http://scrapy.org/ http://scrapy.org/, for example, which is a scraping framework that uses lxml and xpath by default.

bmelton 16y ago
I don't know if it was edited after your post, but it lists Scrapy right next to Beautiful Soup.

il 16y ago

I have a question: Does scraping like this execute Javascript on the scraped page? Am I able to access the output of Javascript/AJAX on that page? As far as I know this is impossible with any other server-side scraping technology. If so, that would be amazingly useful for a couple of my side projects, much easier than parsing their Javascript code and extracting the info I need.

robinduckett 16y ago
You'd have to parse the page seperately and run each piece of in line scripts / linked scripts in a sandbox which can talk to jsdom, but it could be done.
- _delirium 16y ago
  Sounds like a useful general-purpose library someone could put together: server-side execution of HTML-embedded js, with some sort of configurable sandbox (e.g. decide whether you want to let it call out to the internet or not). All the components are available, but seems like the field is open for an all-in-one solution that gets the defaults and edge cases right. I imagine Google has something at least close to that internally, given that they've dropped hints here and there about their ability to crawl post-js-processed pages, but unless I've missed it, I don't think they've released anything (my guess is because of the arms-race issue, with shady sites trying to use js to mask certain content from googlebot).
  - fizx 16y ago
    Selenium Remote Control lets you drive a real browser from the scripting language of your choice. xvfb let's you run headless. You really don't need anything else.
  - il 16y ago
    Second this idea. Nothing like this exists currently except for a couple buggy poorly documented dead projects. Selenium might be OK for usability testing a single site, but it's useless for large scale multithreaded crawling and scraping and doesn't scale. I, for one, would definitely pay good money for an easy library/API to scrape JS-heavy sites without the overhead of a full browser running macros.

robinduckett 16y ago

Hey guys. The Nodejitsu team and Marak (http://www.github.com/Marak http://www.github.com/Marak), the guy behind Nodejitsu are perma-banned from HN and can't respond to your queries. He sends his regards, and if you'd like to contact him visit the #Node.js IRC channel @ Freenode

fizx 16y ago

This reminds me, I ported the core ideas of the parsley scraping language to jQuery. http://github.com/fizx/pquery#readme http://github.com/fizx/pquery#readme

deleted 16y ago

[deleted]

wmil 16y ago
HN deletes posts to try to keeps things on topic. It's heavy handed, but there aren't a lot of ways to stop flame wars and pun threads.
- pygy_ 16y ago
  I apologize. I deleted my former post, because the censorship accusations were based on an error on my side, I had been reading another comment thread related to the author of the post[1] and mixed things up. [1] http://news.ycombinator.com/item?id=1448309 http://news.ycombinator.com/item?id=1448309

knowtheory 16y ago

The article reads "The challenge with using these libraries is that they all have their own quirks that can make working with HTML, CSS and Javascript challenging." And that's true only if you only want to do page manipulation in Javascript. I'm perfectly happy with my page manipulation in Ruby w/ Nokogiri. Here's an example: (code formatting on HN sucks, so it's on my blog, apologies) http://blog.knowtheory.net/post/1074676060/xml-manipulation-in-6-lines-of-ruby http://blog.knowtheory.net/post/1074676060/xml-manipulation-...

deleted 16y ago

[deleted]

tcarnell 16y ago

Interesting, when I built http://cQuery.com http://cQuery.com (Content Query Engine), I investigated a number of options html parsing and content extraction options. I had played with Rhino and John Resigs env.js (http://ejohn.org/blog/bringing-the-browser-to-the-server/ http://ejohn.org/blog/bringing-the-browser-to-the-server/) to run jQuery server-side. For portability, performance and flexability I finally settled for writing my own HTML parser and CSS selection engine from scratch.

jfager 16y ago

Ignoring the drama: my current favorite scraping combo is NekoHtml underneath Scala's completely kickass combo of pattern matching and XML literals.