3 ms·
Google is quite scrupulous about robots.txt. There's plenty of hypocrisy at that company but this doesn't strike me as a case of it.
by samatman 6y ago
Google is quite scrupulous about robots.txt.
There's plenty of hypocrisy at that company but this doesn't strike me as a case of it.
- decasteve 6y agoScrupulous? Nobody is going to disallow / on Google's user-agent. I'd suggest respecting Google's robots.txt as much as Google respects our DNT settings.