5 ms·

Using YQL to grab HN links

54 points by chaosmachine 18y ago

chaosmachine 18y ago
The query is: select * from html where url="http://news.ycombinator.com/" and xpath='//tr/td/a[substring(@href,1,4)="http"][@href!="http://ycombinator.com"]' You can play with it yourself here (needs yahoo login): http://developer.yahoo.com/yql/console/ http://developer.yahoo.com/yql/console/
- Sephr 18y ago
  YQL isn't that great server-side until they add an `Access-Control-Allow-Origin: *` header. I'm pretty disappointed that they completely disallow Javascript access.
  - spullara 18y ago
    It offers JSONP instead so you can access any YQL content via Javascript in a browser via script nodes.
    - Sephr 18y ago
      You still have to give Yahoo! complete control with that method. With XHR, Yahoo! has no control over your webpage.
      - spullara 18y ago
        It only it works in a few browsers but doesn't seem like it would hurt anything to add it. Thanks for the suggestion!
        sachinag 18y ago
        I love how HN is the only site where "[x] would be cool" turns into a feature request implemented by the vendor.
        spullara 18y ago
        Nah, we do the same thing for things suggested on Twitter :)
- thorax 18y ago
  You know, honestly, I wasn't grokking much about how cool YQL could be until I saw this example. This helps a lot. Thanks.
- henning 18y ago
  I don't get it. Don't queries like that suffer from the same problems as normal screen scraping?
  - jfarmer 18y ago
    Yeah, they're obviously brittle, but it's baby steps, y'know? I think it's pretty crazy that you can now scrape well-marked pages with a SQL-like syntax.
rjurney 18y ago
Just spent 20 minutes trying to grab all article links from a newspaper's website based on a url pattern. Failed. The docs could use some work. Anyone know how to do this?
- hsuresh 18y ago
  which newspaper website was it?
- tectonic 18y ago
  For another approach, with a structure editor, check out http://parselets.com http://parselets.com
ghempton 18y ago
Does it first convert the page to valid xhtml and then perform the xpath or does it rely on the website to be well-marked?
- simonw 18y ago
  It works against not-well-formed markup.
yeahit 18y ago
If you use Linux or some other Unix, you can also do it with standard Unix tools: wget -O- news.ycombinator.com | grep -o http[^\"]* Personally, I prefer curl because it writes to stdout per default: curl news.ycombinator.com | grep -o http[^\"]* (After posting this, i noticed that HN cuts * signs at the end of a message. So I have to add this text here, or the last * would not be displayed.)
- sarp 18y ago
  This shows all urls on HN including images etc. whereas the original post demonstrates retrieving only linked articles
  - yeahit 18y ago
    You can filter that with another grep for example: wget -O- news.ycombinator.com | grep -o 'title"><a href="[^"]*' | grep -o http.*
3ds 18y ago
in python it would be like this (get BeautifulSoup first): -- import urllib2 from BeautifulSoup import BeautifulSoup ychtml = urllib2.urlopen('http://news.ycombinator.com/).read() for tdtitle in BeautifulSoup(ychtml).findAll("td", "title"): print tdtitle.a
found_dead 18y ago
I actually attended the talk on YQL at Barcamp Portland and it seems really powerful. You are basically able to use Yahoo's cloud servers and huge internet pipe for free with this service.

54 points by chaosmachine 18y ago

The query is: select * from html where url="http://news.ycombinator.com/" and xpath='//tr/td/a[substring(@href,1,4)="http"][@href!="http://ycombinator.com"]' You can play with it yourself here (needs yahoo login): http://developer.yahoo.com/yql/console/ http://developer.yahoo.com/yql/console/

Sephr 18y ago
YQL isn't that great server-side until they add an `Access-Control-Allow-Origin: *` header. I'm pretty disappointed that they completely disallow Javascript access.
- spullara 18y ago
  It offers JSONP instead so you can access any YQL content via Javascript in a browser via script nodes.
  - Sephr 18y ago
    You still have to give Yahoo! complete control with that method. With XHR, Yahoo! has no control over your webpage.
    - spullara 18y ago
      It only it works in a few browsers but doesn't seem like it would hurt anything to add it. Thanks for the suggestion!
      - sachinag 18y ago
        I love how HN is the only site where "[x] would be cool" turns into a feature request implemented by the vendor.
        spullara 18y ago
        Nah, we do the same thing for things suggested on Twitter :)
thorax 18y ago
You know, honestly, I wasn't grokking much about how cool YQL could be until I saw this example. This helps a lot. Thanks.
henning 18y ago
I don't get it. Don't queries like that suffer from the same problems as normal screen scraping?
- jfarmer 18y ago
  Yeah, they're obviously brittle, but it's baby steps, y'know? I think it's pretty crazy that you can now scrape well-marked pages with a SQL-like syntax.

rjurney 18y ago

Just spent 20 minutes trying to grab all article links from a newspaper's website based on a url pattern. Failed. The docs could use some work. Anyone know how to do this?

hsuresh 18y ago
which newspaper website was it?
tectonic 18y ago
For another approach, with a structure editor, check out http://parselets.com http://parselets.com

ghempton 18y ago

Does it first convert the page to valid xhtml and then perform the xpath or does it rely on the website to be well-marked?

simonw 18y ago
It works against not-well-formed markup.

yeahit 18y ago

If you use Linux or some other Unix, you can also do it with standard Unix tools: wget -O- news.ycombinator.com | grep -o http[^\"]* Personally, I prefer curl because it writes to stdout per default: curl news.ycombinator.com | grep -o http[^\"]* (After posting this, i noticed that HN cuts * signs at the end of a message. So I have to add this text here, or the last * would not be displayed.)

sarp 18y ago
This shows all urls on HN including images etc. whereas the original post demonstrates retrieving only linked articles
- yeahit 18y ago
  You can filter that with another grep for example: wget -O- news.ycombinator.com | grep -o 'title"><a href="[^"]*' | grep -o http.*

3ds 18y ago

in python it would be like this (get BeautifulSoup first): -- import urllib2 from BeautifulSoup import BeautifulSoup ychtml = urllib2.urlopen('http://news.ycombinator.com/).read() for tdtitle in BeautifulSoup(ychtml).findAll("td", "title"): print tdtitle.a

found_dead 18y ago

I actually attended the talk on YQL at Barcamp Portland and it seems really powerful. You are basically able to use Yahoo's cloud servers and huge internet pipe for free with this service.