IHMO, preventing site scraping is really really hard. There are couple of startups that offer products that claim to stop site scraping.[1] 1. Ajaxified pagina

IHMO, preventing site scraping is really really hard. There are couple of startups that offer products that claim to stop site scraping.[1] 1. Ajaxified paginated data -> With Firebug looking at XHR requests, one can easily reverse engineer the ajax call and extract data directly from the json - in fact you are making the scraper's life easier as you will probably be storing the information in a structured manner in the json string. Or one could use a more sophisticated tool like MITMProxy to study how requests are made. If you somehow managed to implement a highly obfuscated method of ajax requests (re: microsoft aspx.net), there is always selenium to get through them. 2. Randomize template output -> You are going to annoy users if you are going to display a different template altogether. If its just randomizing div and class ids, one can write clever xpath expressions or css selectors to circumvent this. Or worse case there are always the ever reliable regex expressions. 3. HoneyPot -> Scrapers only crawl pages that they are specifically looking for. A good scraper only runs through pages he wants to scrape. Nothing more. This is probably the least effective strategy. 4. Write data to images on the fly -> Use an OCR api to decode them! 5. Alternatives -> Putting in a login screen is also not effective as not only will that annoy users, it can easily be circumvented by using selenium or passing the cookie/session information to the scraping script. Blacklisting ips is not going to a very effective strategy. With tons of free proxies, Tor and cloud based services (especially PiCloud[2] - which offers a scraping optimized instance!!), ip blocking can easily be circumvented. Best strategy would be display corrupted content or start throwing CAPTCHAs if you sense a large number of requests coming from a particular IP. But once again you may want to do some sort of machine learning on server logs based on the various ips and the specific urls visited and build a model that could predict if a particular user is a bot or a human before you start throwing fake data or captchas. Just to be on the safe side so that you don't annoy anyone. [1] http://spider.io http://spider.io and http://blockscraping http://blockscraping [2] http://www.picloud.com/pricing/ http://www.picloud.com/pricing/ s1 instance plan