2 ms·
I'm surprised the Wayback Machine doesn't account for that by figuring "well our archive of this predates the current robots.txt, so let's not remove it".
by yellowapple 8y ago
I'm surprised the Wayback Machine doesn't account for that by figuring "well our archive of this predates the current robots.txt, so let's not remove it".
- markjgraham 8y agoYou don't need to be surprised. We do account for what you reference.