You can use cloudfare but it's a small roadblock. I can still crawl that. Also you can do frontend rendering, it's a bit larger roadblock but you can use phant

You can use cloudfare but it's a small roadblock. I can still crawl that. Also you can do frontend rendering, it's a bit larger roadblock but you can use phantomJS or something to crawl that. IIRC there is a php framework that mutate your front end code but I'm not sure if it does it enough to stop a generalized xpath... Also I used to work for company where they employ people full time for crawling. It will even notified the company if crawler stopped working so they can update their crawler...