3 ms·
If anyone's about to mirror the stuff on his web sites, it might be wise to act quickly. Wayback Machine always checks the current robots.txt for a given domain
by cpach 17y ago
If anyone's about to mirror the stuff on his web sites, it might be wise to act quickly. Wayback Machine always checks the current robots.txt for a given domain, so it's possible to block files that have already been archived.