4 ms·

My project: CSVGet -- Get structured data from sites as CSV

52 points by fizx 17y ago

sgrove 17y ago
Wow, incredibly cool. I did the same thing with collection of curl / grep / sed / awk, and it was awful. I later redid it with some python library, and then with hpricot, and most recently with scrubyt. Each step was a little bit better, but I really should have been looking towards making a more generalized solution like this. Well done!
tectonic 17y ago
Use http://selectorgadget.com http://selectorgadget.com to help make the selectors, too.
- hadley 17y ago
  Wow - fantastic!
- vbar 17y ago
  But why mess with the selectors by hand? http://search.cpan.org/~vbar/HTML-ListScraper-0.05/ http://search.cpan.org/~vbar/HTML-ListScraper-0.05/ can discover the structure automatically (well, sometimes :-) - some pages just aren't regular enough, but it does work on HN, for example)...
Aschwin 17y ago
Oh my, this rocks. What would be even cooler is a library to use these functions. That would beat me to it, since I was thinking about a general data interface for websites too. If I could implement your effort into my code (PHP) that would be so awesome. Anyhow, nice idea though.
- fizx 17y ago
  http://github.com/fizx/parsley/tree/master http://github.com/fizx/parsley/tree/master is a C library which represents the core of the parsing functionality. A PHP binding is quite possible, and indeed, there already are bindings for Python and Ruby. In fact, I might just go look at how PHP bindings are done in general...
skorgu 17y ago
A flag to output JSON would be nifty, then you could pass it through a filter [1] and get interweb-grep on steroids. [1] http://goessner.net/articles/JsonPath/ http://goessner.net/articles/JsonPath/
- fizx 17y ago
  Um, you got it. :) I just added a "jsonget" binary to the package. http://gist.github.com/177304 http://gist.github.com/177304
  - skorgu 17y ago
    Now that's service!
hadley 17y ago
This is fantastic. I do a load of data scraping of the web and this (plus selector gadget) is going to make my life sooooo much easier.
mc 17y ago
The real magic behind this gem is another one called Parsley, which is an astounding library for screen scraping. http://github.com/fizx/parsley/tree/master http://github.com/fizx/parsley/tree/master They've even built an entire app on top of the library: http://parselets.com/ http://parselets.com/

52 points by fizx 17y ago

Wow, incredibly cool. I did the same thing with collection of curl / grep / sed / awk, and it was awful. I later redid it with some python library, and then with hpricot, and most recently with scrubyt. Each step was a little bit better, but I really should have been looking towards making a more generalized solution like this. Well done!

tectonic 17y ago

Use http://selectorgadget.com http://selectorgadget.com to help make the selectors, too.

hadley 17y ago
Wow - fantastic!
vbar 17y ago
But why mess with the selectors by hand? http://search.cpan.org/~vbar/HTML-ListScraper-0.05/ http://search.cpan.org/~vbar/HTML-ListScraper-0.05/ can discover the structure automatically (well, sometimes :-) - some pages just aren't regular enough, but it does work on HN, for example)...

Aschwin 17y ago

Oh my, this rocks. What would be even cooler is a library to use these functions. That would beat me to it, since I was thinking about a general data interface for websites too. If I could implement your effort into my code (PHP) that would be so awesome. Anyhow, nice idea though.

fizx 17y ago
http://github.com/fizx/parsley/tree/master http://github.com/fizx/parsley/tree/master is a C library which represents the core of the parsing functionality. A PHP binding is quite possible, and indeed, there already are bindings for Python and Ruby. In fact, I might just go look at how PHP bindings are done in general...

skorgu 17y ago

A flag to output JSON would be nifty, then you could pass it through a filter [1] and get interweb-grep on steroids. [1] http://goessner.net/articles/JsonPath/ http://goessner.net/articles/JsonPath/

fizx 17y ago
Um, you got it. :) I just added a "jsonget" binary to the package. http://gist.github.com/177304 http://gist.github.com/177304
- skorgu 17y ago
  Now that's service!

hadley 17y ago

This is fantastic. I do a load of data scraping of the web and this (plus selector gadget) is going to make my life sooooo much easier.

mc 17y ago

The real magic behind this gem is another one called Parsley, which is an astounding library for screen scraping. http://github.com/fizx/parsley/tree/master http://github.com/fizx/parsley/tree/master They've even built an entire app on top of the library: http://parselets.com/ http://parselets.com/