4 ms·

I haven't put up a robots.txt in years, they are utterly pointless, IMHO. Even google doesn't honor them(nor did any of the others, like Alexa, for many years n

I haven't put up a robots.txt in years, they are utterly pointless, IMHO. Even google doesn't honor them(nor did any of the others, like Alexa, for many years now). Read you web server logs, you'll see googlebot crawling pages it should not, if it were honoring your robots.txt.

spc476 7y ago
Have you checked the IPs to make sure they're actually Google? There's a strong incentive to fake being Google.
- mtnGoat 7y ago
  yes they IPs were owned by them, they even indexed pages they were asked not to. :x