3 ms·
It is a glaring flaw. It's meant a lot of sites ended up wiped out of the archive (or at least made inaccessible) simply because their domain has expired and th
by CM30 9y ago
It is a glaring flaw. It's meant a lot of sites ended up wiped out of the archive (or at least made inaccessible) simply because their domain has expired and the domain squatters blocked the empty domain from being indexed.
The solution (which the Internet Archive really needs to implement) is to look at the domain registration data or something, and then only remove content if the same owner updated the robots.txt file. If not, then just disallow archiving any new content, since the new domain owner usually has no right to decide what happens to the old site content.