3 ms·
This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this
by inovica 13y ago
This looks really good. We're mainly Python focused and have been working on a tool to try to 'train' a crawler to extract specific elements of a page. As this is a crawling thread I hope you don't mind me asking some advice on where to take it from here :)
Here's how it currently works:
1) It has a queue of domains that I have pre-processed. For the initial purposes I've restricted it to pages that I think are ecommerce based on $ signs, add to cart/basket type links etc
2) There is a visual tool that I then use to select certain parts of the page - eg price, product, image etc. I save these out as xpaths
3) Once I have done one URL I send a crawler to that domain and extract other pages that fit the profile of an ecommerce page and try to use the same mapping as number 2 above to extract the data
I have done a small video to show it in action:
http://www.screencast.com/t/riB3iiVMiSk http://www.screencast.com/t/riB3iiVMiSk
I'm not sure if I'm doing this the right way. If a site/page changes structure then I may have to re-map the data. I was hoping that someone would have some pointers for me in terms of any other ways to do this. Also with Javascript-heavy sites I've had some problems
If anyone has any knowledge of screen scraping, where it can be done more automatically, I'd really appreciate a steer!
- regularfry 13y agoI've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful. One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity (http://celerity.rubyforge.org/ http://celerity.rubyforge.org/) with a hacked-on Mechanize API to do the rewriting, which let me handle JS-heavy pages almost as easily as static HTML ones. My original inspiration for this was _why's Mousehole (the source for which is here: https://github.com/evaryont/mousehole https://github.com/evaryont/mousehole, I've got no idea if it runs on recent Rubies). The proxy also gives you somewhere to raise an alert if, all of a sudden, your scraping fails because of an upstream change. One tool I always intended to make some use of, but never got round to, was Ariel: http://ariel.rubyforge.org/ http://ariel.rubyforge.org/. It looks like it ought to be able to totally remove the need to manually extract xpaths.
- inovica 13y agoThanks for this. I'll check these out
- Walkman 13y agoCheck out Scrapy http://scrapy.org/ http://scrapy.org/