4 ms·

Link rot is a serious problem: http://www.gwern.net/Archiving%20URLs#link-rot http://www.gwern.net/Archiving%20URLs#link-rot >In a 2003 experiment, Fetterly et

Link rot is a serious problem: http://www.gwern.net/Archiving%20URLs#link-rot http://www.gwern.net/Archiving%20URLs#link-rot >In a 2003 experiment, Fetterly et al. discovered that about one link out of every 200 disappeared each week from the Internet. McCown et al. (2005) discovered that half of the URLs cited in D-Lib Magazine articles were no longer accessible 10 years after publication [the irony!], and other studies have shown link rot in academic literature to be even worse (Spinellis, 2003, Lawrence et al., 2001). Nelson and Allen (2002) examined link rot in digital libraries and found that about 3% of the objects were no longer accessible after one year. >Bruce Schneier remarks that one friend experienced 50% linkrot in one of his pages over less than 9 years (not that the situation was any better in 1998), and that his own blog posts link to news articles that go dead in days; the Internet Archive has estimated the average lifespan of a Web page at 100 days. A Science study looked at articles in prestigious journals; they didn’t use many Internet links, but when they did, 2 years later ~13% were dead. The French company Linterweb studied external links on the French Wikipedia before setting up their cache of French external links, and found - back in 2008 - already 5% were dead. (The English Wikipedia has seen a 2010-2011 spike from a few thousand dead links to ~110,000 out of ~17.5m live links.) The dismal studies just go on and on and on (and on). Even in a highly stable, funded, curated environment, link rot happens anyway. For example, about 11% of Arab Spring-related tweets were gone within a year (even though Twitter is - currently - still around).

idlewords 12y ago
My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.
- sillysaurus3 12y ago
  You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?
  - idlewords 12y ago
    https://blog.pinboard.in/2014/08/researching_link_rot/ https://blog.pinboard.in/2014/08/researching_link_rot/
    - sillysaurus3 12y ago
      To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection. You chose to include everyone's private bookmarks in your research without asking their consent? What? I will publish some aggregate information about what I find, and use it to seek glory, and persuade people to sign up for archiving. But I won't release anything that could lead back to specific users or links. There is roughly a boatload of evidence that anonymized datasets can be deanonymized in unexpected ways. Even if you don't release any anonymized datasets, it's really not good that you decided to take such liberties with people's private links in the first place.
      - idlewords 12y ago
        Why would I need consent to study the global link rot rate? Publishing it reveals no information about users, either individually or in the aggregate. I've made an effort to let anyone who wants to opt out of the research, because I know people can have strong feelings about privacy. I agree with you that publishing an 'anonymized' dataset would be a violation of privacy guarantees. I wouldn't even do it for public bookmarks.
        sillysaurus3 12y ago
        "Private" means "private," not "private unless Maciej wants to study them." I didn't get an email saying "There's a chance I might select your private bookmarks and examine them." A blog post doesn't count when you're messing with people's private data. Certainly it should be opt-in, and not opt-out? You're doing this for a noble purpose, but for what it's worth, this is the first time my trust in you has ever felt violated. People have entrusted you with years worth of private data, and you just asked, "Why should I ask permission to study their private links?" Actually, as far as I can tell, your comment seems to implicitly assume that you already have consent to examine all private links, and that asking consent would only be necessary if you were planning on publishing something that might reveal some of their private links. Isn't that the opposite of privacy?
- Houshalter 12y ago
  That's consistent with most of those studies. The next paragraph does an interesting experiment showing a very conservative rate of 3% per year: >My specific target date is 2070, 60 years from now. As of 10 March 2011, gwern.net has around 6800 external links (with around 2200 to non-Wikipedia websites)4. Even at the lowest estimate of 3% annual linkrot, few will survive to 2070. If each link has a 97% chance of surviving each year, then the chance a link will be alive in 2070 is 0.972070−2011=0.16 (or to put it another way, an 84% chance any given link will die). The 95% confidence interval for such a binomial distribution says that of the 2200 non-Wikipedia links, ~336-394 will survive to 20705. If we try to predict using a more reasonable estimate of 50% linkrot, then an average of 0 links will survive (0.502070−2011×2200=1.735×10−16×2200≃0). It would be a good idea to simply assume that no link will survive.

idlewords 12y ago

My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.

sillysaurus3 12y ago
You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?
- idlewords 12y ago
  https://blog.pinboard.in/2014/08/researching_link_rot/ https://blog.pinboard.in/2014/08/researching_link_rot/
  - sillysaurus3 12y ago
    To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection. You chose to include everyone's private bookmarks in your research without asking their consent? What? I will publish some aggregate information about what I find, and use it to seek glory, and persuade people to sign up for archiving. But I won't release anything that could lead back to specific users or links. There is roughly a boatload of evidence that anonymized datasets can be deanonymized in unexpected ways. Even if you don't release any anonymized datasets, it's really not good that you decided to take such liberties with people's private links in the first place.
    - idlewords 12y ago
      Why would I need consent to study the global link rot rate? Publishing it reveals no information about users, either individually or in the aggregate. I've made an effort to let anyone who wants to opt out of the research, because I know people can have strong feelings about privacy. I agree with you that publishing an 'anonymized' dataset would be a violation of privacy guarantees. I wouldn't even do it for public bookmarks.
      - sillysaurus3 12y ago
        "Private" means "private," not "private unless Maciej wants to study them." I didn't get an email saying "There's a chance I might select your private bookmarks and examine them." A blog post doesn't count when you're messing with people's private data. Certainly it should be opt-in, and not opt-out? You're doing this for a noble purpose, but for what it's worth, this is the first time my trust in you has ever felt violated. People have entrusted you with years worth of private data, and you just asked, "Why should I ask permission to study their private links?" Actually, as far as I can tell, your comment seems to implicitly assume that you already have consent to examine all private links, and that asking consent would only be necessary if you were planning on publishing something that might reveal some of their private links. Isn't that the opposite of privacy?
Houshalter 12y ago
That's consistent with most of those studies. The next paragraph does an interesting experiment showing a very conservative rate of 3% per year: >My specific target date is 2070, 60 years from now. As of 10 March 2011, gwern.net has around 6800 external links (with around 2200 to non-Wikipedia websites)4. Even at the lowest estimate of 3% annual linkrot, few will survive to 2070. If each link has a 97% chance of surviving each year, then the chance a link will be alive in 2070 is 0.972070−2011=0.16 (or to put it another way, an 84% chance any given link will die). The 95% confidence interval for such a binomial distribution says that of the 2200 non-Wikipedia links, ~336-394 will survive to 20705. If we try to predict using a more reasonable estimate of 50% linkrot, then an average of 0 links will survive (0.502070−2011×2200=1.735×10−16×2200≃0). It would be a good idea to simply assume that no link will survive.