3 ms·
Yeah, they really ought to work from the version of robots.txt archived along with a given snapshot. If the legal owner of the content wants it taken down, the
by JoshTriplett 13y ago
Yeah, they really ought to work from the version of robots.txt archived along with a given snapshot. If the legal owner of the content wants it taken down, they can submit an explicit request for that.