4 ms·
Apropos archive.org: I recently found out they interpret a missing robots.txt as access denied. I wonder if that's a mistake or on purpose.
by didibc 11y ago
Apropos archive.org: I recently found out they interpret a missing robots.txt as access denied. I wonder if that's a mistake or on purpose.
- abstractbeliefs 11y agoGenerally, media without a license is considered "All rights reserved" by default. Archive.org may be choosing to interpret no robots.txt similarly out of courtesy (better not archive something sensitive by accident) or to minimise legal liability.
- richardbrevig 11y agoAs of April 18, 2015...I know that's not true. They archived a site of mine that does not have a robots.txt So it must be a very new policy, because prior to that there are long histories of my sites that don't have robots.txt files being archived.