3 ms·
Incorrect. To configure your robots.txt to not be crawled at all use: User-agent: * Disallow: / Allow indexing of everything with: User-agent: * Disa
by blauwbilgorgel 14y ago
Incorrect. To configure your robots.txt to not be crawled at all use:
User-agent: *
Disallow: /
Allow indexing of everything with:
User-agent: *
Disallow:
It seems they are at this very moment struggling to change things.
- rodion_89 14y agoYou are totally right. Disregard everything I said.