If you need to do form stepping, you need to look at something like Perl's Mechanize package. (Ruby has one too.) Spend time reading articles related to Mechan

If you need to do form stepping, you need to look at something like Perl's Mechanize package. (Ruby has one too.) Spend time reading articles related to Mechanize. Your resulting code is going to be fairly terse, so you don't really need to spend much time worrying about making a generic crawler.