18 ms·
And of course, remember that robots.txt is only a signal to benevolent bots which respect it. If you have secrets to keep, don't put them online in the first pl
by zeta0134 6y ago
And of course, remember that robots.txt is only a signal to benevolent bots which respect it. If you have secrets to keep, don't put them online in the first place.
- archgoon 6y agoOr properly authenticate (and audit) access.
- hinkley 6y agoLooking at my Splunk logs and then asking a lot of questions, I have learned that there are a LOT of not so benevolent bots that must be tolerated anyway. Benevolence is a continuum.
- blacksmith_tb 6y agoIn fact robots.txt is a list of things a nefarious crawler will absolutely want to examine - no need to know those paths when they're all laid out for you!