You could do a reverse lookup on crawler IP addresses to make sure that they match the User Agent. Google PTR records will end in google.com or googlebot.com. M

You could do a reverse lookup on crawler IP addresses to make sure that they match the User Agent. Google PTR records will end in google.com or googlebot.com. Maybe compare with the list at http://chceme.info/ips/ http://chceme.info/ips/ You probably can’t prevent this altogether, but you can probably do a lot to raise the cost of this type of crawling. The exception might be if there are people out to copy your content specifically and will actually tailor their crawlers to your countermeasures.