3 ms·
They only disable access, not delete. There was once a court case where the defendant got the court to compel the plaintiff to alter their robots.txt in order t
by wtallis 14y ago
They only disable access, not delete. There was once a court case where the defendant got the court to compel the plaintiff to alter their robots.txt in order to allow the defendant to gather evidence from the archive. (Apparently, the Archive managed to convince the court that manually producing the info hidden in the archive would be too onerous.)