2 ms·
I have a specific exclusion in my robots.txt file, and also a cron-scheduled grep of my logs to see if anything actually hits it. I don't care if they do or th
by michaelcampbell 6y ago
I have a specific exclusion in my robots.txt file, and also a cron-scheduled grep of my logs to see if anything actually hits it. I don't care if they do or they don't, but it's a way for me to exclude specific bots that don't honor my robots.txt file.