3 ms·
My impression archive.org follows whatever directions are in robots.txt. If those directions allow a site to be archived, they do it, otherwise they don't. That
by joe_the_user 5y ago
My impression archive.org follows whatever directions are in robots.txt. If those directions allow a site to be archived, they do it, otherwise they don't. That's from looking at not-archived sites, ezboard, which mention in another post.