5 ms·

About Linkrot: Proportion of Working Links on Pinboard.in, 1997-2011

118 points by chl 15y ago

patio11 15y ago
There is an easy way to increase your detection of redirects and parked pages: make two requests, one to the real URL and one to a URL which is intentionally broken. (example.com/i-am-a-link and example.com/fklsdfasdifo for example) Run a heuristic for difference on the resulting content. This won't catch all of them, particularly if you use a really naive heuristic that can't deal with e.g. ads changing, but it's a heck of a lot quicker than comparing manually.
- hcho 15y ago
  I suspect that's what Google does too. I see lots of example.com/fklsdfasdifo type requests from Big G in my logs.
  - bauchidgw 15y ago
    if you see a lot of them go to webmaster tools, if you see them there too its not some kind of test but some other reasone, mostly their shitty js parsinf, which treats anything with an / as a relative url...
- benhoyt 15y ago
  Yeah, that's a good method to detect "soft 404s". I wrote a Python library for DecentURL to do this some time ago: http://blog.brush.co.nz/2008/01/soft404s/ http://blog.brush.co.nz/2008/01/soft404s/ (Ironically, the original paper I got the idea from seems to be a dead link: http://www.tomkinshome.com/andrew/papers/decay/final/p444-baryossef.htm http://www.tomkinshome.com/andrew/papers/decay/final/p444-ba...)
- ible 15y ago
  I had a project/startup working on dealing with link rot for a while. It would not just tell you a link was 404ing, but recognize when the content of the page had changed significantly and let you know. The fun part was to automatically recommend a good replacement page from within the site, nearby, or the internet archive. Based on a quick test it seemed that it would take an site owner about 5-10 minutes per link to find a good replacement once they knew it was broken . That's fine for a personal portfolio with 50 links, but for a site like Boing Boing getting all the broken links working again looked like full time work for a year. I'm curious if you have many outbound links in your 'scalable content'? Do you spend much time maintaining them?
- true_religion 15y ago
  I just want to point out that this won't really work if the site tries to redirect you to a search engine. Naturally broken links will come up with all kinds of search results, but deliberately broken links will come up with "None".
bmm6o 15y ago
"Links appear to die at a steady rate (they don't have a half life), and you can expect to lose about a quarter of them every seven years." Is this self-contradictory, or is just poor wording of his findings?
- yahelc 15y ago
  Half life isn't a steady rate; it's exponential gradual decay. For example, if the element yahelium had a half-life of a year, it would degrade by 1/2 every year. So, the degradation during the first year would be twice the degradation of the second year. That's not linear.
- deleted 15y ago
  [deleted]
- pyninja 15y ago
  What's not clear about it? They die at a steady rate of 25% per 7 years.
  - bahadden 15y ago
    So they have a quarter-life of 7 years?
  - bmm6o 15y ago
    As reemrevnivek explains, the answer to "25% of what?" is the issue here. An exponential decay can be described with the same words.
    - pyninja 15y ago
      Sorry, I didn't think of the second interpretation.
    - erikpukinskis 15y ago
      There's a freaking graph. Are we really being this pedantic that we can't understand what he's saying?
ugh 15y ago
I think it's about time that some government or billionaire throws a few millions at an internet archive project. The Internet Archive is nice but more regular snapshots with a wider coverage would be something I'm certain future historians would love to get their hands on (and they will hate us if we don't do it).
- tokenadult 15y ago
  I think it's about time that some government or billionaire throws a few millions at an internet archive project. It may be that one or two governments have already done that. You are, of course, referring to a publicly accessible Internet archive. As for what a benevolent millionaire (it wouldn't have to be a full billionaire for this to start up) could fund, pg has suggested, "There is room to do to Wikipedia what Wikipedia did to Britannica." http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html It's interesting that pg thought then that Wikipedia's problem is excessive deletionism, while I (after being a registered Wikipedian and working on various articles) think that Wikipedia's problem is lack of thorough research to prepare article content. http://strategy.wikimedia.org/wiki/Wikimedia_Movement_Strategic_Plan_Summary/Improve_Quality http://strategy.wikimedia.org/wiki/Wikimedia_Movement_Strate... Whatever one's opinion of what's wrong with Wikipedia, the best way to prompt improvement in Wikipedia (or replace it, if you prefer that) is to build another site that does some of what Wikipedia does but does it better somehow. That's not easy, not easy at all, but it's not terribly expensive. I have looked at the Wikimedia Foundation financial reports, and building a strong competitor to Wikipedia is a project that is well within the grasp of several individual millionaires, and within the grasp of quite a few nonprofit charitable organizations. A business corporation that can find out a way to monetize a Wikipedia competitor might have a great business opportunity.
  - tmarthal 15y ago
    Fitting, in regards to the original pinboard.in story, is that in the CRM section of the http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html (number 7.) links to an article http://articles.latimes.com/2008/may/29/business/fiw-octopart29 http://articles.latimes.com/2008/may/29/business/fiw-octopar... that has rotted and is now a link to a generic not-found(w/HTTP 200) latimes.com page.
gojomo 15y ago
Users may prune their own bookmarks when they discover the links broken – especially when considering some of the pre-Pinboard systems (like in-browser bookmarking) from which the earliest data in this analysis comes. So I suspect this underestimates link-rot.
- idlewords 15y ago
  Just like it says in the article!
  - gojomo 15y ago
    D'oh! Overlooked that in my quick-read (or seeing as how I chose the same 'prune' word, only noticed it subconsciously). Still an important enough point to pull out for highlight here; I should have used a direct quote.
InclinedPlane 15y ago
Sadly I think technological advances has only accelerated this phenomenon. We've gone from an era of static pages that would require considerable effort to change the overall layout of to CMSes that we can twiddle and upgrade with nary a concern for backward link compatibility. Personally I think it should be a principle of every professional web developer that you just don't break links, period.

118 points by chl 15y ago

There is an easy way to increase your detection of redirects and parked pages: make two requests, one to the real URL and one to a URL which is intentionally broken. (example.com/i-am-a-link and example.com/fklsdfasdifo for example) Run a heuristic for difference on the resulting content. This won't catch all of them, particularly if you use a really naive heuristic that can't deal with e.g. ads changing, but it's a heck of a lot quicker than comparing manually.

hcho 15y ago
I suspect that's what Google does too. I see lots of example.com/fklsdfasdifo type requests from Big G in my logs.
- bauchidgw 15y ago
  if you see a lot of them go to webmaster tools, if you see them there too its not some kind of test but some other reasone, mostly their shitty js parsinf, which treats anything with an / as a relative url...
benhoyt 15y ago
Yeah, that's a good method to detect "soft 404s". I wrote a Python library for DecentURL to do this some time ago: http://blog.brush.co.nz/2008/01/soft404s/ http://blog.brush.co.nz/2008/01/soft404s/ (Ironically, the original paper I got the idea from seems to be a dead link: http://www.tomkinshome.com/andrew/papers/decay/final/p444-baryossef.htm http://www.tomkinshome.com/andrew/papers/decay/final/p444-ba...)
ible 15y ago
I had a project/startup working on dealing with link rot for a while. It would not just tell you a link was 404ing, but recognize when the content of the page had changed significantly and let you know. The fun part was to automatically recommend a good replacement page from within the site, nearby, or the internet archive. Based on a quick test it seemed that it would take an site owner about 5-10 minutes per link to find a good replacement once they knew it was broken . That's fine for a personal portfolio with 50 links, but for a site like Boing Boing getting all the broken links working again looked like full time work for a year. I'm curious if you have many outbound links in your 'scalable content'? Do you spend much time maintaining them?
true_religion 15y ago
I just want to point out that this won't really work if the site tries to redirect you to a search engine. Naturally broken links will come up with all kinds of search results, but deliberately broken links will come up with "None".

bmm6o 15y ago

"Links appear to die at a steady rate (they don't have a half life), and you can expect to lose about a quarter of them every seven years." Is this self-contradictory, or is just poor wording of his findings?

yahelc 15y ago
Half life isn't a steady rate; it's exponential gradual decay. For example, if the element yahelium had a half-life of a year, it would degrade by 1/2 every year. So, the degradation during the first year would be twice the degradation of the second year. That's not linear.
deleted 15y ago
[deleted]
pyninja 15y ago
What's not clear about it? They die at a steady rate of 25% per 7 years.
- bahadden 15y ago
  So they have a quarter-life of 7 years?
- bmm6o 15y ago
  As reemrevnivek explains, the answer to "25% of what?" is the issue here. An exponential decay can be described with the same words.
  - pyninja 15y ago
    Sorry, I didn't think of the second interpretation.
  - erikpukinskis 15y ago
    There's a freaking graph. Are we really being this pedantic that we can't understand what he's saying?

ugh 15y ago

I think it's about time that some government or billionaire throws a few millions at an internet archive project. The Internet Archive is nice but more regular snapshots with a wider coverage would be something I'm certain future historians would love to get their hands on (and they will hate us if we don't do it).

tokenadult 15y ago
I think it's about time that some government or billionaire throws a few millions at an internet archive project. It may be that one or two governments have already done that. You are, of course, referring to a publicly accessible Internet archive. As for what a benevolent millionaire (it wouldn't have to be a full billionaire for this to start up) could fund, pg has suggested, "There is room to do to Wikipedia what Wikipedia did to Britannica." http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html It's interesting that pg thought then that Wikipedia's problem is excessive deletionism, while I (after being a registered Wikipedian and working on various articles) think that Wikipedia's problem is lack of thorough research to prepare article content. http://strategy.wikimedia.org/wiki/Wikimedia_Movement_Strategic_Plan_Summary/Improve_Quality http://strategy.wikimedia.org/wiki/Wikimedia_Movement_Strate... Whatever one's opinion of what's wrong with Wikipedia, the best way to prompt improvement in Wikipedia (or replace it, if you prefer that) is to build another site that does some of what Wikipedia does but does it better somehow. That's not easy, not easy at all, but it's not terribly expensive. I have looked at the Wikimedia Foundation financial reports, and building a strong competitor to Wikipedia is a project that is well within the grasp of several individual millionaires, and within the grasp of quite a few nonprofit charitable organizations. A business corporation that can find out a way to monetize a Wikipedia competitor might have a great business opportunity.
- tmarthal 15y ago
  Fitting, in regards to the original pinboard.in story, is that in the CRM section of the http://ycombinator.com/ideas.html http://ycombinator.com/ideas.html (number 7.) links to an article http://articles.latimes.com/2008/may/29/business/fiw-octopart29 http://articles.latimes.com/2008/may/29/business/fiw-octopar... that has rotted and is now a link to a generic not-found(w/HTTP 200) latimes.com page.

gojomo 15y ago

Users may prune their own bookmarks when they discover the links broken – especially when considering some of the pre-Pinboard systems (like in-browser bookmarking) from which the earliest data in this analysis comes. So I suspect this underestimates link-rot.

idlewords 15y ago
Just like it says in the article!
- gojomo 15y ago
  D'oh! Overlooked that in my quick-read (or seeing as how I chose the same 'prune' word, only noticed it subconsciously). Still an important enough point to pull out for highlight here; I should have used a direct quote.

InclinedPlane 15y ago

Sadly I think technological advances has only accelerated this phenomenon. We've gone from an era of static pages that would require considerable effort to change the overall layout of to CMSes that we can twiddle and upgrade with nary a concern for backward link compatibility. Personally I think it should be a principle of every professional web developer that you just don't break links, period.