3 ms·
I was suggesting Disallow to make sure Google doesn't request it ;) I don't know if any bots look at robots.txt to see potentially interesting URLs. I do when I
by luckylion 7y ago
I was suggesting Disallow to make sure Google doesn't request it ;) I don't know if any bots look at robots.txt to see potentially interesting URLs. I do when I take a better look at sites, but I usually don't qualify as a bot.
My experience is that most bots just hit the usual suspects, /wp-login.php, /phpmyadmin/ etc, regardless whether they are in robots.txt or not.
- IronBacon 7y ago> My experience is that most bots just hit the usual suspects, /wp-login.php, /phpmyadmin/ etc, regardless whether they are in robots.txt or not. Yeah, basically what I see in my logs. To be more clear, the disallow is for a non existent path in the document dir. I somewhat expected to find at least one script to actively crawl it, but it makes sense, as no sane people would put secrets on a website and protect them with a robot.txt... ^__^;