4 ms·
It's really not that hard to get info taken down from IA if you are the site owner. Official blog from 2017 https://blog.archive.org/2017/04/17/robots-txt-meant
by sp332 4y ago
It's really not that hard to get info taken down from IA if you are the site owner. Official blog from 2017 https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... and more recent third-party blog with a survey of several ways to remove content (that you control) from Wayback https://blog.reputationx.com/block-wayback-machine https://blog.reputationx.com/block-wayback-machine
- charcircuit 4y agorobots.txt only prohibits people from using the wayback machine to view the archive. The IA still has it archived. If someone later changes the robots.txt to something more permissive it will put back everything on the wayback machine.
- kappuchino 4y agoWrong. (If you know better than I please cite a credible source). I requested to purge the archive from a domain I have since the former owner put semi legal crap there. Apart from robots text, I was asked to set up one specific page with a hash-like url to signal that I am really, really sure about the purge. Thats all that is needed. The process is described here: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som...
- sp332 4y agoWell there is this example from 2007 https://archive.org/post/133690/robotstxt-only-gives-temporary-removal-of-pages https://archive.org/post/133690/robotstxt-only-gives-tempora...
- charcircuit 4y agoI only have anecdotal evidence from a domain becoming parked with a restricted robots.txt and then it getting unparked and the archive still existing.