14 ms·You have robots.txt standard that you can use to exclude crawling.by kolinko 4y agoYou have robots.txt standard that you can use to exclude crawling.ipaddr 4y agorobots.txt is not a law. It's what google pushed. No one has to obey robots.txt and will often use it to find things you forbid indexingnl 4y agorobots.txt was around before Google