3 ms·
The first page you linked tells the story of how the Wayback Machine once deleted archived content if robots.txt directed it to, but was moving towards not doin
by roryokane 6y ago
The first page you linked tells the story of how the Wayback Machine once deleted archived content if robots.txt directed it to, but was moving towards not doing that anymore. You can also see in GitHub’s robots.txt that URLs matching `/*/tree/` should already have been forbidden, yet the Wayback Machine at one point had https://github.com/github/dmca/tree/565ece486c7c1652754d7b6d2b5ed9cb4097f9d5 https://github.com/github/dmca/tree/565ece486c7c1652754d7b6d... archived. These facts suggest that it is your second theory that is correct—GitHub contacted archive.org to get the archive taken down.