3 ms·
robots.text is not a security tool. It's communication tool that gives advice. Just like sitemaps. If you need to add security (logins) to protect content you
by nabla9 10y ago
robots.text is not a security tool. It's communication tool that gives advice. Just like sitemaps.
If you need to add security (logins) to protect content you don't need to protect you inconvenience users.
- laumars 10y agoI'd already covered the security point replying to another poster (https://news.ycombinator.com/item?id=14163792 https://news.ycombinator.com/item?id=14163792) but just to be clear, I'm absolutely not claiming robots.txt is a security tool. Quite the opposite, I saying following good security and design practices renders the robots.txt file obsolete. Your point about sitemaps helps illustrate that point of mine because having a decent sitemap mitigates the need for Allow lines in robots.txt. It's another feature of the web where robots.txt isn't well equipped to handle and thus there have been other, better, tools built to highlight pages of interest to search engines.