3 ms·
IIRC, if even the robots.txt on a site gets changed, it's applied retroactively to the wayback machine. So older versions won't get shown. I'm quite concerned
by extraduder_ire 3y ago
IIRC, if even the robots.txt on a site gets changed, it's applied retroactively to the wayback machine. So older versions won't get shown.
I'm quite concerned about this for some domains I no longer control.