3 ms·
IA needs to respects robots.txt and they need to make it easier to request data to be removed. Not everyone fully supports the IA mission and they need to resp
by iJohnDoe 5y ago
IA needs to respects robots.txt and they need to make it easier to request data to be removed.
Not everyone fully supports the IA mission and they need to respect that view as much as they respect their supporters.
- smarx007 5y agoThey clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA. Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycombinator.com/item?id=29645135 https://web.archive.org/web/20211222032633/https://news.ycom... oops!
- iJohnDoe 5y agoAre you indicating that everything in the IA was added via a person clicking on “archive this page”? I don’t think that is correct. A lot of it was added via automated methods?
- smarx007 5y agoNot necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://news.ycombinator.com/item?id=29643506 https://news.ycombinator.com/item?id=29643506 For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules. https://archivebox.io https://archivebox.io