45 ms·
Why couldn't they get the sanitized archive they wanted with robots.txt, but just excluding the specific URLs they wanted excluded in robots.txt?
by jrochkind1 9y ago
Why couldn't they get the sanitized archive they wanted with robots.txt, but just excluding the specific URLs they wanted excluded in robots.txt?