2 ms·
I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the cas
by Analemma_ 4y ago
I want archive.org to ignore robots.txt and make it as difficult as possible to remove pages from it; it would be a broken archive tool if this were not the case.
- anonymousiam 4y agoSometimes there's a legitimate need to remove archived content. Back in the early days of running an ISP, I had some users hide links to copyrighted content with "map name" in a hosted image. Eventually the search engines began indexing these links and my net traffic exploded. I was not happy.
- gigel82 4y agoI hope you never accidentally publish private information anywhere on the web.