3 ms·

So the solution I am going to implement soon is taking a tool like ArchiveBox or SinglePage and hosting my own copies of (most) external links, so they will be

by kick 7y ago

So the solution I am going to implement soon is taking a tool like ArchiveBox or SinglePage and hosting my own copies of (most) external links, so they will be cached shortly after linking and can't break. The bandwidth and space will be somewhat expensive, but it'll save me and my readers a ton in the long run. Wouldn't a more ideal solution be archiving via a variety of external (or internal, I guess) sources the first time a link appears on your site, and then after a year automatically switching all links to archived versions? This would kill link rot in its tracks while preserving a lot of the value of links for the people you're linking to, and would cost less in bandwidth costs given the curve of access on old content.

gwern 7y ago
> Wouldn't a more ideal solution be archiving via a variety of external (or internal, I guess) sources the first time a link appears on your site, and then after a year automatically switching all links to archived versions? Yes, by 'shortly' I meant something like 90 days. In my experience, most pages won't change too much 90 days after I add them (I'm thinking particularly of social media-like things), but it's also rare for something to die that quickly. 365+ days, however, would be perilously long. My main concern there is balancing between delaying so long to snapshot that the link dies (thus generating the manual linkrot-repair problem I'm trying to avoid) and being too eager to snapshot and archiving a version which is not done and would mislead a reader. (I also went through all my domains and created a whitelist of domains that my experience suggests are trustworthy or where local mirrors wouldn't be relevant. For example, obviously you don't really need to worry about Arxiv links breaking, or about English Wikipedia pages disappearing - barring the occasional deletionist rampage.) > This would kill link rot in its tracks while preserving a lot of the value of links for the people you're linking to, and would cost less in bandwidth costs given the curve of access on old content. My traffic patterns are different from a blog, so it wouldn't.

by kick 7y ago

gwern 7y ago

> Wouldn't a more ideal solution be archiving via a variety of external (or internal, I guess) sources the first time a link appears on your site, and then after a year automatically switching all links to archived versions? Yes, by 'shortly' I meant something like 90 days. In my experience, most pages won't change too much 90 days after I add them (I'm thinking particularly of social media-like things), but it's also rare for something to die that quickly. 365+ days, however, would be perilously long. My main concern there is balancing between delaying so long to snapshot that the link dies (thus generating the manual linkrot-repair problem I'm trying to avoid) and being too eager to snapshot and archiving a version which is not done and would mislead a reader. (I also went through all my domains and created a whitelist of domains that my experience suggests are trustworthy or where local mirrors wouldn't be relevant. For example, obviously you don't really need to worry about Arxiv links breaking, or about English Wikipedia pages disappearing - barring the occasional deletionist rampage.) > This would kill link rot in its tracks while preserving a lot of the value of links for the people you're linking to, and would cost less in bandwidth costs given the curve of access on old content. My traffic patterns are different from a blog, so it wouldn't.