3 ms·
Nobody follows robots.txt, because every site's robots.txt forbids anybody that isn't google from looking at it. Also, "hiding their identity" is what every si
by marcosdumay 2y ago
Nobody follows robots.txt, because every site's robots.txt forbids anybody that isn't google from looking at it.
Also, "hiding their identity" is what every single browser does since Mosaic changed its name.
- paulryanrogers 2y agoIncluding extra, legacy agents isn't hiding because they include their distinct identifiers too.