Yes, that is exactly what the crawler should do. I had no crawler experience a few months ago but wanted to do the same thing. I didn't know Ruby so I want the

Yes, that is exactly what the crawler should do. I had no crawler experience a few months ago but wanted to do the same thing. I didn't know Ruby so I want the Java route. Now it's trivially easy, I just wrote a crawler last night to pull a bunch of info off hundreds of pages for my next app. First start with how you yourself would get at each page of data. If it all starts from a head page figure out how to get to each link, follow those links and repeat until you are at the data pages. Then point your crawler to the head page and code in the patterns. If the pages are not all linked from a head page but have a url pattern figure out how to pull down the info needed to fill in the url pieces, pull that info down then visit each page filling in the pattern with the data you now have. I can't give any Ruby guidance but if you have general questions on web scraping you can shoot me an email.