3 ms·
From one of my projects: https://www.checkbot.io/robots.txt https://www.checkbot.io/robots.txt Also, it's a really common misunderstanding that robots.txt can
by seanwilson 5y ago
From one of my projects:
https://www.checkbot.io/robots.txt https://www.checkbot.io/robots.txt
Also, it's a really common misunderstanding that robots.txt can be used to keep pages out of search results. You need to use noindex meta tags or noindex headers for that. Robots.txt tells crawlers which pages they can't fetch, but they're still allowed to include those pages in search results if they want.
- Cthulhu_ 5y agoAnd even those are advisory, if you really need to block them, you have to add rules to your webserver of known user agents and IPs (because plenty of indexers will spoof their user agent as well. Google probably does that so they can penalize those that offer different content to search engines than regular users. Doesn't really work though).