3 ms·
Scrupulous? Nobody is going to disallow / on Google's user-agent. I'd suggest respecting Google's robots.txt as much as Google respects our DNT settings.
by decasteve 6y ago
Scrupulous? Nobody is going to disallow / on Google's user-agent.
I'd suggest respecting Google's robots.txt as much as Google respects our DNT settings.