4 ms·
Definitely tune your robots.txt But if you put exact URLs in your robots.txt, then a normal person could pick them up from there and access your content. Make
by chris100 17y ago
Definitely tune your robots.txt
But if you put exact URLs in your robots.txt, then a normal person could pick them up from there and access your content.
Make sure you use a directory structure and forbid all files from the content directory :-)
- andrewljohnson 17y agoIt's also worth pointing out that only the likes of Google and Yahoo will even look at robots.txt. Some other crawler may walk in there, then create pages that link your files, and then Google could link those pages, and it's game over.