3 ms·
I'm working on-and-off on something not totally unrelated and the number of times that a site has been archived, disappeared, but then replaced by a domain park
by ChickeNES 9y ago
I'm working on-and-off on something not totally unrelated and the number of times that a site has been archived, disappeared, but then replaced by a domain parked page with robots.txt restrictions is infuriating. The data is there, but with no way to access it. I understand obeying robots.txt when crawling, but applying it to existing content is just idiotic.