3 ms·
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results. I realize that you are just the messenger and not the prog
by gunapologist99 4y ago
> Warning: Don't use a robots.txt file as a means to hide your web pages from Google search results.
I realize that you are just the messenger and not the progenitor of that policy, so not addressing this to you, but: that is ridiculous. robots.txt is basically useless.
- jeroenhd 4y agoRobots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.