2 ms·
The Wayback machine honors robots.txt retroactively. Which is really unfortunate when a domain lapses and ends up owned by a completely different entity.
by pimlottc 12y ago
The Wayback machine honors robots.txt retroactively. Which is really unfortunate when a domain lapses and ends up owned by a completely different entity.