I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful. One approach

I've done almost exactly this in the past. There's a hell of a lot of fiddling in keeping the xpaths both stable and general enough to be useful. One approach I found absolutely vital was to have a rewriting, caching proxy between the crawler and the upstream site. This proxy allowed me to rewrite the page content into something much simpler for the crawler to get to grips with (RSS or Atom, say). I used Celerity (http://celerity.rubyforge.org/ http://celerity.rubyforge.org/) with a hacked-on Mechanize API to do the rewriting, which let me handle JS-heavy pages almost as easily as static HTML ones. My original inspiration for this was _why's Mousehole (the source for which is here: https://github.com/evaryont/mousehole https://github.com/evaryont/mousehole, I've got no idea if it runs on recent Rubies). The proxy also gives you somewhere to raise an alert if, all of a sudden, your scraping fails because of an upstream change. One tool I always intended to make some use of, but never got round to, was Ariel: http://ariel.rubyforge.org/ http://ariel.rubyforge.org/. It looks like it ought to be able to totally remove the need to manually extract xpaths.