3 ms·
They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ct
by smarx007 5y ago
They clearly explained that they don’t consider archival to be a robotic activity when a person clicks “archive this page”. It’s closer to saving a page with Ctrl+S and uploading it to IA.
Removing data from the web in 2021? Hmm… https://web.archive.org/web/20211222032633/https://news.ycombinator.com/item?id=29645135 https://web.archive.org/web/20211222032633/https://news.ycom... oops!
- iJohnDoe 5y agoAre you indicating that everything in the IA was added via a person clicking on “archive this page”? I don’t think that is correct. A lot of it was added via automated methods?
- smarx007 5y agoNot necessarily via a button but all of the pages were submitted to them by 3rd parties. So, they don't seem to crawl to discover new URIs eagerly, see https://news.ycombinator.com/item?id=29643506 https://news.ycombinator.com/item?id=29643506 For example, I host my own ArchiveBox at home (you get fulltext search as a bonus) and it is configured to submit every URL I save to IA: https://imgur.com/a/Yhnxo1W https://imgur.com/a/Yhnxo1W IA considers that to be manual submission not subject to robots.txt rules. https://archivebox.io https://archivebox.io