3 ms·

How-to stop most people from spidering your site and stealing content

• jacquesm 17y ago
  This blocking strategy is so naive it's not even funny. Your best (or most fanatic) users will be blocked and the smart bot operators will get through. Better to adopt a more sophisticated strategy. - send a cookie with the request, make sure it's present on subsequent requests - embed other media in your pages and check that they are fetched too (make sure these can't be cached) - put a list of all known tor exit nodes and open proxies in a DB somewhere and check against that. You could use IP tables for this or you can handle it in your application layer - put some velocity controls in that aim to make it expensive to fetch sequential ids if your site is set up like that. - sprinkle your site with unclickable links that link to trigger pages that block the ip, or if you want to go sophisticated on your wannabe spiderers store the ip in a database and from then on whenever that IP connects spout complete garbage in large pages that link to other garbage pages. This will cost you some bandwidth though, if your bandwidth costs are lower than the perps this will at some point lead to a reduction in bot activity. - keep checks on the user agents, anything with 'bot' in it that is not on a permitted list can get the same treatment, a list of url fetching libraries is also handy.