3 ms·You could do this using a robots.txt file (assuming the scraper obeys it, of course).by watchdogtimer 6y agoYou could do this using a robots.txt file (assuming the scraper obeys it, of course).