4 ms·
It always has. If your site is publicly available and you don't disallow bots through robots.txt, they can crawl it at any time. Even if the site is "pre-launch
by jorams 6y ago
It always has. If your site is publicly available and you don't disallow bots through robots.txt, they can crawl it at any time. Even if the site is "pre-launch", because that doesn't mean anything on its own.
- zeta0134 6y agoAnd of course, remember that robots.txt is only a signal to benevolent bots which respect it. If you have secrets to keep, don't put them online in the first place.
- archgoon 6y agoOr properly authenticate (and audit) access.
- hinkley 6y agoLooking at my Splunk logs and then asking a lot of questions, I have learned that there are a LOT of not so benevolent bots that must be tolerated anyway. Benevolence is a continuum.
- blacksmith_tb 6y agoIn fact robots.txt is a list of things a nefarious crawler will absolutely want to examine - no need to know those paths when they're all laid out for you!
- ghaff 6y agoI'd just add that, while major players like the Internet Archive do respect robots.txt, it's essentially just a flag that depends on people voluntarily respecting it. If a site is publicly available but you don't want people to find it, you're just depending of security through obscurity.
- AnonHP 6y agoInternet Archive stopped respecting robots.txt since 2017. See https://boingboing.net/2017/04/22/internet-archive-to-ignore-rob.html https://boingboing.net/2017/04/22/internet-archive-to-ignore...
- f311a 6y agoWebarchive completely ignores robots for a few years now. They did it on purpose.
- deleted 6y ago[deleted]
- AnonHP 6y agoI have serious issues with this and the fact that site owners have to email a human support team in archive.org to be excluded.