3 ms·
IIRC Google do automated checks on robots.txt and report in webmaster tools if you did something that looks crazy. https://qph.fs.quoracdn.net/main-qimg-002d1f
by pbhjpbhj 7y ago
IIRC Google do automated checks on robots.txt and report in webmaster tools if you did something that looks crazy.
https://qph.fs.quoracdn.net/main-qimg-002d1f819e1bfbbd14fa2de937b489d0 https://qph.fs.quoracdn.net/main-qimg-002d1f819e1bfbbd14fa2d... shows some of the interface, but I seem to recall getting notified in webmaster tools when I messed up the robots.txt on a particular site.