5 ms·
Hah! tectonic and I applied to YC with almost exactly this in 2009?! We went as far as building a browser-based IDE-like environment for generating these, and
by fizx 14y ago
Hah! tectonic and I applied to YC with almost exactly this in 2009?!
We went as far as building a browser-based IDE-like environment for generating these, and a language called parsley for expressing the scrapes. If you're interested in this, you could check out some of our related open source libraries:
Edit: I just open-sourced the scraping wiki site we created here: https://github.com/fizx/parselets_com https://github.com/fizx/parselets_com
http://selectorgadget.com http://selectorgadget.com
https://github.com/fizx/parsley https://github.com/fizx/parsley
https://github.com/fizx/parsley-ruby https://github.com/fizx/parsley-ruby
https://github.com/fizx/pyparsley https://github.com/fizx/pyparsley
https://github.com/fizx/csvget https://github.com/fizx/csvget
> cat hn.let
{
"headlines":[{
"title": ".title a",
"link": ".title a @href",
"comments": "match(.subtext a:nth-child(3), '\\d+')",
"user": ".subtext a:nth-child(2)",
"score": "match(.subtext span, '\\d+')",
"time": "match(.subtext, '\\d+\\s+\\w+\\s+ago')"
}]
}
> csvget --directory-prefix=./data -A "/x" -w 5 --parselet=hn.let http://news.ycombinator.com/
> head data/headlines.csv
comments,title,time,link,score,user
4,Simpson's paradox: why mistrust seemingly simple statistics,2 hours ago,http://en.wikipedia.org/wiki/Simpson%27s_paradox,41,waldrews
67,America's unjust sex laws,2 hours ago,http://www.economist.com/opinion/displaystory.cfm?story_id=14165460,59,MikeCapone
23,Buy somebody lunch,3 hours ago,http://www.whattofix.com/blog/archives/2009/08/buy-somebody-lu.php,58,DanielBMarkham
- sc00ter 14y agoOh! Selector gadget is wonderful, and I still use it to this day. It's so easy to forget that there are real people behind the tools we use - so fizx and tectonic, I thank you!
- christiangenco 14y agoSeconded - selector gadget is one of the best scraping tools around. You've saved me many a frustrated evenings.
- tectonic 14y agoThanks for the kind words! Maybe this attention will push me to update the code and make a new screencast.
- sathish316 14y agoDo you mind if i use selector gadget in APIfy? I use firebug and firefinder to get selectors. But i can't expect everyone to have the same tools.
- fizx 14y agoEverything mentioned is MIT-licensed, and like many OSS authors, we would love any ideas or code in there to be widely used in virtually any capacity. Also, if you're in or near SF, I'd be happy to get coffee sometime.
- sathish316 14y agoThanks for suggesting SelectorGadget. I ended up building an in-browser firebug+firefinder clone to make the app even more easier to use. Please check out Preview webpage for any API
- tluyben2 14y agoI was going to suggest here that you do. And opensource your creation on github so we can all improve it.
- tectonic 14y agoPlease do use selectorgadget! And if you'd like to push anything back, or chat about parsing in general, send me a message. I have a more advanced branch of SG that can generate better selectors, but I haven't pushed it out yet. It's all in the repo.
- tluyben2 14y agoSelectorgadget is great; without checking out your creations in real life, did you consider taping selectorgadget to a proxy so you can scrape sites and store the paths you found in one go? That would massively enhance the process imho :) Maybe Apify can do that, but I hope they put that in github as well; i'm not a great fan of closed source/cloud development tools.