Yeah as a scraper I'd say that at most all these suggestions would do is make me turn to selenium/greasemonkey instead of mechanize/wget/httrack. Selenium is th

Yeah as a scraper I'd say that at most all these suggestions would do is make me turn to selenium/greasemonkey instead of mechanize/wget/httrack. Selenium is the bomb when people try to get fancy preventing scraping, how exactly are they supposed to detect the difference between a browser and a browser? Getting banned is not a big deal, plenty of IPs & proxies out there. EC2 is your best friend as you can automate the IP recycling. Even Facebook/Twitter accounts are almost free. Even the randomization wouldn't be particularly difficult to circumvent just save the page and then use a genetic algorithm with tunable parameters for the randomization, select the parameters that yield the most/best records. What I'd actually fear is a system that just silently corrupted the records once scraping was detected, especially if it was intermittent, eg. 10-75% of records on a page are bogus and only every few pages. Or they started displaying the records as images (but I'm guessing they want Google juice)