3 ms·
To Google: User-agent: * Disallow: / threatens their ability to index and serve ads against the open web, which is really what they mean.
by ethank 14y ago
To Google:
User-agent: *
Disallow: /
threatens their ability to index and serve ads against the open web, which is really what they mean.
- jlarocco 14y agoNot really. robots.txt has always been based on a good faith. All the reputable companies abide by it, but I don't doubt for a second they'd stop reading it if it started affecting their bottom line.