Ouch, the Google ban stick does not sound fun. An alternative for spidering high-risk avenues like that might be 80legs.com, in essence a legal/opt-in crawling

Ouch, the Google ban stick does not sound fun. An alternative for spidering high-risk avenues like that might be 80legs.com, in essence a legal/opt-in crawling botnet =P I've used them when they were in open beta and the service was free -- now you need to pay, although it's pretty cheap for smallish jobs.