12 ms·

Wikipedia and Internet Archive partner to fix 1M broken links on Wikipedia

494 points by The_ed17 10y ago

jacquesm 10y ago
In the long term the internet archive will likely be the major supplier of references to Wikipedia. Webpages don't live forever, hopefully the internet archive does. It's an extremely valuable resource, the archive and wikipedia are amongst the most valuable digital assets we have.
- HCIdivision17 10y ago
  I'll include the obligatory donation link below for what is effectively an internet utility. [0] Also, take a look at downloading a copy of Wikipedia! You can get a full download with images for around 100GB (last I checked, which was about two years ago). It's great for if you ever think you'll need some technical info while away from live internet. I keep a copy on a hard drive that boots into linux, just in case (maybe I'll be on site with a customer and need some engineering notes - BAM taken care of!) [0] https://archive.org/donate/ https://archive.org/donate/ EDIT: I used XOWA, and I do not keep the wiki up to date, really. Note that the entire wiki history is huge, but a reasonably current snapshot is manageable (~100GB or so). [~] http://xowa.org/home/wiki/Help/Download_XOWA.html http://xowa.org/home/wiki/Help/Download_XOWA.html
  - theGimp 10y ago
    Thank you for posting the donation link. I have donated in the past and always think about donating when the Internet Archive is mentioned. I just set up a monthly donation amount, and I encourage you to do the same if you believe in the Internet Archive's mission!
  - protomikron 10y ago
    Actually I think 100G is not that much, if you consider what you get. Furthermore some time ago I estimated that you could store the entire world's tiles (raster data in 10m resolution) and OSM data on a SSD (<500G). If you think about it, that might be a game changer. Maybe it might make sense to ship e.g. smart phones with wikipedia, OSM and satellite photo (e.g. Sentinel-2 is free data) on disk. Granted, 10m resolution is not the state-of-the-art in aerial imagery - which is around ~0.1m (this means 10^4 times more data) - but it is reasonable to detect buildings and combined with OSM's vector data you have practically Google Maps in your pocket.
    - toomuchtodo 10y ago
      > If you think about it, that might be a game changer. Maybe it might make sense to ship e.g. smart phones with wikipedia, OSM and satellite photo (e.g. Sentinel-2 is free data) on disk. I don't think this is a sensible use of local phone storage, but I do think you'll see a lot more P2P edge cache nodes if/when IPFS takes off (OSM tiles are already served on the IPFS network). It would be a simple matter of picking a VPS provider or hardware colo provider near expected heavy use, launching IPFS, and having it pin the relevant content locally.
      - anc84 10y ago
        How is caching handled for those tiles in IPFS? At OSM.org they get rerendered when the underlying data changes.
        toomuchtodo 10y ago
        https://github.com/ipfs/faq/issues/16 https://github.com/ipfs/faq/issues/16
    - iotscale 10y ago
      If we're considering just the availability, Google Maps has an option to cache a region for offline use, which works spectacularly good - given that you get your antenna on a good signal every once in a while.
      - takingflac 10y ago
        I mostly use this because limited data plan, makes it use a lot less data if you already downloaded the area you are currently in before hand while you were still on wifi.
- cmdrfred 10y ago
  I have a pretty substantial amount of storage for things I have collected over the years. Podcasts, YouTube videos, Gifs even stuff from back when Flash animations were popular. All organized and curated. Because it's private I don't really have to worry about copyright laws as well. 6 disks in a raid 10 array with 2 hot spares. It's rather cumbersome now but hopefully the tech will get good enough to be able to maintain itself for a few hundred years before I die. I know some of that stuff will be lost to time, some of it you already can't find online. I call it the ark. I hope some day down the road I can blow some historians mind. Perhaps, long from now, they might quote internet comments from sites like HN like we quote the Greek philosophers. "cmdrfred the elder once said...". I should make a habit of ripping the front page or so and comments for the ark.
  - schoen 10y ago
    > Perhaps, long from now, they might quote internet comments from sites like HN like we quote the Greek philosophers. "cmdrfred the elder once said..." Or Talmudic claims of authority in Judaism: https://duckduckgo.com/?q=%22taught+in+the+name+of+rabbi%22&ia=web https://duckduckgo.com/?q=%22taught+in+the+name+of+rabbi%22&... https://duckduckgo.com/?q=%22said+in+the+name+of+rabbi%22&ia=web https://duckduckgo.com/?q=%22said+in+the+name+of+rabbi%22&ia... Muslim ahadith also have the phenomenon of the chain of narration where people declare the provenance of the teaching: https://en.wikipedia.org/wiki/Hadith_studies#Sanad_and_matn https://en.wikipedia.org/wiki/Hadith_studies#Sanad_and_matn
- danso 10y ago
  I think a case could be made that the Archive should be the reference by default for citations -- that is, when a URL is cited, the Archive's snapshot API is triggered, and the archive URL is used as the direct link with the original URL being the "backup". Sites/pages change so much that it's more helpful to point the average user to what a page looked like when the reference was accessed and included. It may rob the original URL some traffic, but Wikipedia isn't meant to be a link-aggregator for external content. The obvious roadblocks are: - Websites, even content-focused ones, can be so complicated in terms of JS that the Archive might not capture it accurately. - The Archive's policy is to honor robots.txt and other no-archive directives.
  - nothrabannosir 10y ago
    It may rob the original URL some traffic, That is an advantage. It reduces extraneous incentives to post links, which should be only about the information they provide and not about page views. At wikipedia's scale, and sensitivity to information purity, that's relevant. See also the rel="nofollow" decision from a few years back.
    - eli 10y ago
      It'd be a real shame if publishers decide to opt out of the archive to reclaim "lost" pageviews.
      - iotscale 10y ago
        It's probably good that it's possible to opt-out of being archived but what annoys me is a useful domain changing hands and the new owners deciding to wipe its history.
        thomasahle 10y ago
        Surely that's not possible?? And there is no way to find the old archive, even under an alias?
- The_ed17 10y ago
  I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their hands full.
  - dredmorbius 10y ago
    I had to migrate my archived articles off Readability prior to the OEL at the end of September. I'd only used the service for a couple of years, and hit about a 5% bitrot rate, this on fairly significant articles. One of the more curious cases was CSIRO (Australia's national science and research organisation) which seems to have not only deliberately purged a fair amount of data (Graham Turner's work specifically), but has a robots.txt in place which blocks archival by TIA. That strikes me as ... downright curious.
    - caf 10y ago
      Which CSIRO site is that with the robots.txt? (it's not their main site). I'm thinking that could be something to have an Opposition Senator bring up in Senate Estimates.
    - toomuchtodo 10y ago
      Could you reply with CSIRO links blocked by robots.txt? I run my own instance of ArchiveTeam's ArchiveBot (which archives links provided regardless of robots.txt), and would be happy to put the content into cold storage.
      - dredmorbius 10y ago
        I'll need to check my Readability dump.
        toomuchtodo 10y ago
        No rush; I'll bookmark this thread to check two weeks from now.
- b2600 10y ago
  Wikipedia and The Internet Archive are examples to me of what makes the internet awe inspiring. (I can throw in Google search too, not Google Inc). Data (sometimes imperfect) is one query away on any internet connected device. Truly amazing in a way.
pmiller2 10y ago
This is a huge step forward for Wikipedia as an authoritative source of information. Glad to see this happening. :) OT: I considered applying to the Internet Archive last time I was looking for work, but their office is too hard to commute to coming from the East Bay. :(
- brudgers 10y ago
  I agree that it's a big step forward for encyclopedias. Not just as a 'source of truth' but also in terms of automating away a lot of the routine editorial maintenance that needs to happen at Wikipedia's scale.
sengork 10y ago
Internet Archive should look into distributed models such as IPFS for storage of the archived sites.
- toomuchtodo 10y ago
  The IPFS team is working with the Internet Archive on this.
  - sengork 10y ago
    Excellent, it may be a better way to provide spare bandwidth/storage similar to their https://archive.org/details/archiveteam-warrior https://archive.org/details/archiveteam-warrior
    - toomuchtodo 10y ago
      For sure. ArchiveTeam has explored providing mirrors of the entire Archive [+], but IPFS is a perfect fit for the task. [+] http://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://www.archiveteam.org/index.php?title=INTERNETARCHIVE.B...
youdontknowtho 10y ago
Wow. I really love the internet archive as a project. This is a great usage. Looking forward to see how that will work out. I wonder if they will publish a list of replaced links after the fact?
shortformblog 10y ago
Excellent news. Should note that today is the 20th anniversary of the Internet Archive: https://blog.archive.org/2016/10/26/making-the-web-more-reliable-20-years-and-counting/ https://blog.archive.org/2016/10/26/making-the-web-more-reli...
eriknstr 10y ago
The archive.is guy provides mirrors of rotten links to Wikipedia also, although not as the result of any official agreement with Wikipedia, just on his own initiative, which I think was nice of him. Enclyclopedia Dramatica is generally not a reputable source of truth, being the site that it is, but while looking for some more information on archive.is mirroring of links from Wikipedia articles, I found an article on ED that I found interesting. It is heavily advocating one side of the story but at least it backs it up with some links, which is rather seldom on ED (most links on ED usually go to other pages on ED in my experience). https://encyclopediadramatica.se/Archive.is https://encyclopediadramatica.se/Archive.is
- necessity 10y ago
  It's amazing to me how Wikipedia ends up being a reasonably good website with such a cancerous community behind it.
  - caf 10y ago
    Perhaps this is the genius of Wikipedia - it keeps many of those of a certain type of personality occupied amongst themselves while using the energy of their machinations to produce a product of wider social good.
  - TorKlingberg 10y ago
    What are you referring to?
    - corobo 10y ago
      I'm not necessarily agreeing with the OP here and I don't even know what the community is like but this seems a decent page to start[1]. I do like how Wikipedia keeps a page on it's own controversies - I mean it makes sense, but I like that they're open about it. [1] https://en.wikipedia.org/wiki/List_of_Wikipedia_controversies https://en.wikipedia.org/wiki/List_of_Wikipedia_controversie...
  - cooper12 10y ago
qwertyuiop924 10y ago
I'm really glad this is happening. Wikipedia needs to clean up their broken links, and this could help the archive get a wider sampling of websites, so as to preserve more data. Websites going offline is a huge problem. For example, the now-famous thread from which sleepsort originated (on 4chan's /prog/ textboard) isn't archived anywhere: textboard threads are immortal, so nobody thought to archive any threads until dis.4chan.org went down for good. Thankfully, some bright spark managed to save the sqlite databases for most of the boards on dis to the Internet Archive, so I was able to track down the thread eventually.
45h34jh53k4j 10y ago
(red heart)(yellow heart)(green heart)(blue heart) Internet Archive (red heart)(yellow heart)(green heart)(blue emoji) There are fewer more noble pursuits than archiving the sum of human knowledge.
caf 10y ago
It'd be great if StackOverflow approached the Internet Archive about doing the same for their broken links, too.
alecco 10y ago
On a side note, it makes me very sad how Wikipedia editors are often pushing some political agenda. I'm relying on it for less and less topics. Clearly nothing that can be affected by US politics or SJW-style controversies.
torrent-of-ions 10y ago
Why does the headline says "to fix 1M broken links" but the article says it's already been done?
- cooper12 10y ago
  Yeah that's a bit confusing. I'd attribute it to the "press release" nature of Wikimedia's blog where they mean that they have already partnered with the Internet Archive and are announcing it after the fact.
raverbashing 10y ago
One corner case that exists: a content is linked on Wikipedia, this content is taken down due to a copyright violation (I suppose Archive.org would be asked to take the content down)
- sp332 10y ago
  Archive.org will take content down for certain reasons, but they have a pretty broad copyright exemption as a non-profit archive.
felipesabino 10y ago
As I have clicked in several broken links already, I am wondering how many, absolute number or in percentage, per article are likely to be broken I might be way off, but doesn't 1M seems like a low number for wikipedia size? What is that in percentage of total number of links? Does anyone know?
h1d 10y ago
What's blocking Wikipedia to just archive the referenced pages on edit? It would be far more reliable than depending on Internet Archive when it may not have the page archived and more likely the time of the archive would differ from the time it was referenced. It would cost some more disk space and bandwidth, which of course is already pressuring them but in turn would greatly improve usability and reliability.
- digi_owl 10y ago
  Likely some interpretation of how Wikipedia is not to be a primary source.
ideonexus 10y ago
This whole discussion reminds me of how all MySpace content was destroyed in a rash corporate decision years ago. Just like that, five years of the most popular social networking site on the World Wide Web and all its history were wiped out: http://activehistory.ca/2013/06/myspace-is-cool-again-too-bad-they-destroyed-history-along-the-way/ http://activehistory.ca/2013/06/myspace-is-cool-again-too-ba... Unfortunately, the Internet Archive was only able to get the non-logged-in version of the site. All those loud, obnoxious profile pages users spent endless hours working on? We only have oral histories now to remember them.
- smsm42 10y ago
  I wish I could get all my old horrible homepages back. There was time you'd have to torture me to admit I had anything to do with that, but now I would probably be proud of them again. It's history now.

494 points by The_ed17 10y ago

In the long term the internet archive will likely be the major supplier of references to Wikipedia. Webpages don't live forever, hopefully the internet archive does. It's an extremely valuable resource, the archive and wikipedia are amongst the most valuable digital assets we have.

HCIdivision17 10y ago
I'll include the obligatory donation link below for what is effectively an internet utility. [0] Also, take a look at downloading a copy of Wikipedia! You can get a full download with images for around 100GB (last I checked, which was about two years ago). It's great for if you ever think you'll need some technical info while away from live internet. I keep a copy on a hard drive that boots into linux, just in case (maybe I'll be on site with a customer and need some engineering notes - BAM taken care of!) [0] https://archive.org/donate/ https://archive.org/donate/ EDIT: I used XOWA, and I do not keep the wiki up to date, really. Note that the entire wiki history is huge, but a reasonably current snapshot is manageable (~100GB or so). [~] http://xowa.org/home/wiki/Help/Download_XOWA.html http://xowa.org/home/wiki/Help/Download_XOWA.html
- theGimp 10y ago
  Thank you for posting the donation link. I have donated in the past and always think about donating when the Internet Archive is mentioned. I just set up a monthly donation amount, and I encourage you to do the same if you believe in the Internet Archive's mission!
- protomikron 10y ago
  Actually I think 100G is not that much, if you consider what you get. Furthermore some time ago I estimated that you could store the entire world's tiles (raster data in 10m resolution) and OSM data on a SSD (<500G). If you think about it, that might be a game changer. Maybe it might make sense to ship e.g. smart phones with wikipedia, OSM and satellite photo (e.g. Sentinel-2 is free data) on disk. Granted, 10m resolution is not the state-of-the-art in aerial imagery - which is around ~0.1m (this means 10^4 times more data) - but it is reasonable to detect buildings and combined with OSM's vector data you have practically Google Maps in your pocket.
  - toomuchtodo 10y ago
    > If you think about it, that might be a game changer. Maybe it might make sense to ship e.g. smart phones with wikipedia, OSM and satellite photo (e.g. Sentinel-2 is free data) on disk. I don't think this is a sensible use of local phone storage, but I do think you'll see a lot more P2P edge cache nodes if/when IPFS takes off (OSM tiles are already served on the IPFS network). It would be a simple matter of picking a VPS provider or hardware colo provider near expected heavy use, launching IPFS, and having it pin the relevant content locally.
    - anc84 10y ago
      How is caching handled for those tiles in IPFS? At OSM.org they get rerendered when the underlying data changes.
      - toomuchtodo 10y ago
        https://github.com/ipfs/faq/issues/16 https://github.com/ipfs/faq/issues/16
  - iotscale 10y ago
    If we're considering just the availability, Google Maps has an option to cache a region for offline use, which works spectacularly good - given that you get your antenna on a good signal every once in a while.
    - takingflac 10y ago
      I mostly use this because limited data plan, makes it use a lot less data if you already downloaded the area you are currently in before hand while you were still on wifi.
cmdrfred 10y ago
I have a pretty substantial amount of storage for things I have collected over the years. Podcasts, YouTube videos, Gifs even stuff from back when Flash animations were popular. All organized and curated. Because it's private I don't really have to worry about copyright laws as well. 6 disks in a raid 10 array with 2 hot spares. It's rather cumbersome now but hopefully the tech will get good enough to be able to maintain itself for a few hundred years before I die. I know some of that stuff will be lost to time, some of it you already can't find online. I call it the ark. I hope some day down the road I can blow some historians mind. Perhaps, long from now, they might quote internet comments from sites like HN like we quote the Greek philosophers. "cmdrfred the elder once said...". I should make a habit of ripping the front page or so and comments for the ark.
- schoen 10y ago
  > Perhaps, long from now, they might quote internet comments from sites like HN like we quote the Greek philosophers. "cmdrfred the elder once said..." Or Talmudic claims of authority in Judaism: https://duckduckgo.com/?q=%22taught+in+the+name+of+rabbi%22&ia=web https://duckduckgo.com/?q=%22taught+in+the+name+of+rabbi%22&... https://duckduckgo.com/?q=%22said+in+the+name+of+rabbi%22&ia=web https://duckduckgo.com/?q=%22said+in+the+name+of+rabbi%22&ia... Muslim ahadith also have the phenomenon of the chain of narration where people declare the provenance of the teaching: https://en.wikipedia.org/wiki/Hadith_studies#Sanad_and_matn https://en.wikipedia.org/wiki/Hadith_studies#Sanad_and_matn
danso 10y ago
I think a case could be made that the Archive should be the reference by default for citations -- that is, when a URL is cited, the Archive's snapshot API is triggered, and the archive URL is used as the direct link with the original URL being the "backup". Sites/pages change so much that it's more helpful to point the average user to what a page looked like when the reference was accessed and included. It may rob the original URL some traffic, but Wikipedia isn't meant to be a link-aggregator for external content. The obvious roadblocks are: - Websites, even content-focused ones, can be so complicated in terms of JS that the Archive might not capture it accurately. - The Archive's policy is to honor robots.txt and other no-archive directives.
- nothrabannosir 10y ago
  It may rob the original URL some traffic, That is an advantage. It reduces extraneous incentives to post links, which should be only about the information they provide and not about page views. At wikipedia's scale, and sensitivity to information purity, that's relevant. See also the rel="nofollow" decision from a few years back.
  - eli 10y ago
    It'd be a real shame if publishers decide to opt out of the archive to reclaim "lost" pageviews.
    - iotscale 10y ago
      It's probably good that it's possible to opt-out of being archived but what annoys me is a useful domain changing hands and the new owners deciding to wipe its history.
      - thomasahle 10y ago
        Surely that's not possible?? And there is no way to find the old archive, even under an alias?
The_ed17 10y ago
I've been an editor on Wikipedia for years, and it's simply amazing how many web pages I referenced in 2008–09 have disappeared. Digital archivists have their hands full.
- dredmorbius 10y ago
  I had to migrate my archived articles off Readability prior to the OEL at the end of September. I'd only used the service for a couple of years, and hit about a 5% bitrot rate, this on fairly significant articles. One of the more curious cases was CSIRO (Australia's national science and research organisation) which seems to have not only deliberately purged a fair amount of data (Graham Turner's work specifically), but has a robots.txt in place which blocks archival by TIA. That strikes me as ... downright curious.
  - caf 10y ago
    Which CSIRO site is that with the robots.txt? (it's not their main site). I'm thinking that could be something to have an Opposition Senator bring up in Senate Estimates.
  - toomuchtodo 10y ago
    Could you reply with CSIRO links blocked by robots.txt? I run my own instance of ArchiveTeam's ArchiveBot (which archives links provided regardless of robots.txt), and would be happy to put the content into cold storage.
    - dredmorbius 10y ago
      I'll need to check my Readability dump.
      - toomuchtodo 10y ago
        No rush; I'll bookmark this thread to check two weeks from now.
b2600 10y ago
Wikipedia and The Internet Archive are examples to me of what makes the internet awe inspiring. (I can throw in Google search too, not Google Inc). Data (sometimes imperfect) is one query away on any internet connected device. Truly amazing in a way.

pmiller2 10y ago

This is a huge step forward for Wikipedia as an authoritative source of information. Glad to see this happening. :) OT: I considered applying to the Internet Archive last time I was looking for work, but their office is too hard to commute to coming from the East Bay. :(

brudgers 10y ago
I agree that it's a big step forward for encyclopedias. Not just as a 'source of truth' but also in terms of automating away a lot of the routine editorial maintenance that needs to happen at Wikipedia's scale.

sengork 10y ago

Internet Archive should look into distributed models such as IPFS for storage of the archived sites.

toomuchtodo 10y ago
The IPFS team is working with the Internet Archive on this.
- sengork 10y ago
  Excellent, it may be a better way to provide spare bandwidth/storage similar to their https://archive.org/details/archiveteam-warrior https://archive.org/details/archiveteam-warrior
  - toomuchtodo 10y ago
    For sure. ArchiveTeam has explored providing mirrors of the entire Archive [+], but IPFS is a perfect fit for the task. [+] http://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://www.archiveteam.org/index.php?title=INTERNETARCHIVE.B...

youdontknowtho 10y ago

Wow. I really love the internet archive as a project. This is a great usage. Looking forward to see how that will work out. I wonder if they will publish a list of replaced links after the fact?

shortformblog 10y ago

Excellent news. Should note that today is the 20th anniversary of the Internet Archive: https://blog.archive.org/2016/10/26/making-the-web-more-reliable-20-years-and-counting/ https://blog.archive.org/2016/10/26/making-the-web-more-reli...

eriknstr 10y ago

The archive.is guy provides mirrors of rotten links to Wikipedia also, although not as the result of any official agreement with Wikipedia, just on his own initiative, which I think was nice of him. Enclyclopedia Dramatica is generally not a reputable source of truth, being the site that it is, but while looking for some more information on archive.is mirroring of links from Wikipedia articles, I found an article on ED that I found interesting. It is heavily advocating one side of the story but at least it backs it up with some links, which is rather seldom on ED (most links on ED usually go to other pages on ED in my experience). https://encyclopediadramatica.se/Archive.is https://encyclopediadramatica.se/Archive.is

necessity 10y ago
It's amazing to me how Wikipedia ends up being a reasonably good website with such a cancerous community behind it.
- caf 10y ago
  Perhaps this is the genius of Wikipedia - it keeps many of those of a certain type of personality occupied amongst themselves while using the energy of their machinations to produce a product of wider social good.
- TorKlingberg 10y ago
  What are you referring to?
  - corobo 10y ago
    I'm not necessarily agreeing with the OP here and I don't even know what the community is like but this seems a decent page to start[1]. I do like how Wikipedia keeps a page on it's own controversies - I mean it makes sense, but I like that they're open about it. [1] https://en.wikipedia.org/wiki/List_of_Wikipedia_controversies https://en.wikipedia.org/wiki/List_of_Wikipedia_controversie...
- cooper12 10y ago

qwertyuiop924 10y ago

I'm really glad this is happening. Wikipedia needs to clean up their broken links, and this could help the archive get a wider sampling of websites, so as to preserve more data. Websites going offline is a huge problem. For example, the now-famous thread from which sleepsort originated (on 4chan's /prog/ textboard) isn't archived anywhere: textboard threads are immortal, so nobody thought to archive any threads until dis.4chan.org went down for good. Thankfully, some bright spark managed to save the sqlite databases for most of the boards on dis to the Internet Archive, so I was able to track down the thread eventually.

45h34jh53k4j 10y ago

(red heart)(yellow heart)(green heart)(blue heart) Internet Archive (red heart)(yellow heart)(green heart)(blue emoji) There are fewer more noble pursuits than archiving the sum of human knowledge.

caf 10y ago

It'd be great if StackOverflow approached the Internet Archive about doing the same for their broken links, too.

alecco 10y ago

On a side note, it makes me very sad how Wikipedia editors are often pushing some political agenda. I'm relying on it for less and less topics. Clearly nothing that can be affected by US politics or SJW-style controversies.

torrent-of-ions 10y ago

Why does the headline says "to fix 1M broken links" but the article says it's already been done?

cooper12 10y ago
Yeah that's a bit confusing. I'd attribute it to the "press release" nature of Wikimedia's blog where they mean that they have already partnered with the Internet Archive and are announcing it after the fact.

raverbashing 10y ago

One corner case that exists: a content is linked on Wikipedia, this content is taken down due to a copyright violation (I suppose Archive.org would be asked to take the content down)

sp332 10y ago
Archive.org will take content down for certain reasons, but they have a pretty broad copyright exemption as a non-profit archive.

felipesabino 10y ago

As I have clicked in several broken links already, I am wondering how many, absolute number or in percentage, per article are likely to be broken I might be way off, but doesn't 1M seems like a low number for wikipedia size? What is that in percentage of total number of links? Does anyone know?

h1d 10y ago

What's blocking Wikipedia to just archive the referenced pages on edit? It would be far more reliable than depending on Internet Archive when it may not have the page archived and more likely the time of the archive would differ from the time it was referenced. It would cost some more disk space and bandwidth, which of course is already pressuring them but in turn would greatly improve usability and reliability.

digi_owl 10y ago
Likely some interpretation of how Wikipedia is not to be a primary source.

ideonexus 10y ago

This whole discussion reminds me of how all MySpace content was destroyed in a rash corporate decision years ago. Just like that, five years of the most popular social networking site on the World Wide Web and all its history were wiped out: http://activehistory.ca/2013/06/myspace-is-cool-again-too-bad-they-destroyed-history-along-the-way/ http://activehistory.ca/2013/06/myspace-is-cool-again-too-ba... Unfortunately, the Internet Archive was only able to get the non-logged-in version of the site. All those loud, obnoxious profile pages users spent endless hours working on? We only have oral histories now to remember them.

smsm42 10y ago
I wish I could get all my old horrible homepages back. There was time you'd have to torture me to admit I had anything to do with that, but now I would probably be proud of them again. It's history now.