3 ms·

I never understood this desire to "back up" the internet. It's an impossible endeavour and will only become more incomplete as the net grows. And 99,999% of th

by moe 12y ago

I never understood this desire to "back up" the internet. It's an impossible endeavour and will only become more incomplete as the net grows. And 99,999% of the data "backed up" will never be looked at. What's the point?

ghuntley 12y ago
Well, I personally spent the better half of tonight recovering a bunch of old websites of mine from the IA using wget. IA is one of the most important initiatives/projects on the internet, links are not meant to change but they do and when it happens knowledge & history erodes. Remember the 2004 Indian Ocean earthquake and tsunami? It was the first major event that provided mainstream credibility to the blogosphere, before this disaster consumers turned to news publications for information. For the first time ever news publications did not have the info and were desperate for content, any. TV crews flew into the effected areas and started purchasing suitcases/video/still cameras sight unseen off shell shocked tourists whom were too traumatized to remember if they had captured anything. It was also the last major disaster that occurred before the social revolution - YouTube, Flickr, Facebook, smart-phones and mainstream adoption of broadband. Back then the main way for the general public to share information was via email (which had limits of 10mb) and if they were technical or knew someone technical on their 100mb of free hosting provided by their ISP. Now imagine you're a student, academic researcher, production assistant and you have been tasked researching what transpired. As of right now, the majority of content from 2004/the disaster is no-longer available on the internet and only accessible via the IA. (nb: I was the guy behind http://web.archive.org/web/20050711081524/http://www.waveofdestruction.org/ http://web.archive.org/web/20050711081524/http://www.waveofd... which was the first emergency crisis live/micro blog. For what Google now has a team of volunteers doing during a crisis I did it myself with a single pentium 4 cpu, 4GB of ram, a single IDE hard drive, a ramdrive, a 100mbit uplink and a week without sleep. WOD will be relaunching within the next couple of weeks and was one of the sites recovered from the IA.)
- moe 12y ago
  knowledge & history erodes There surely is history worth preserving. It's just this "compulsive hoarding"-approach that seems ultimately futile to me. The internet itself inherently preserves the knowledge that at least one person cares enough about to put it on a webserver. Pretty much everything of even minor relevance should be referenced by Wikipedia by now. IMHO the internet itself is the "Internet Archive". Remember the 2004 Indian Ocean earthquake and tsunami? To be honest, no. But Wikipedia seems to have that event pretty well covered; http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_and_tsunami http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_an...
  - ghaff 12y ago
    There is some balance between "compulsive hoarding" and preservation. I actually agree that it's an ultimately futile effort--and not very fruitful--to obsess over preserving every bit of text, audio, and video that's ever been posted to the Internet. On the other hand, preserving the content of Usenet or even Geocities do strike me as worth some disk space and won't happen organically for the most part. And the sort of people who have the mindset to do that preservation probably inherently have a bit of the compulsive hoarder about them. On balance, I think it's a good thing.
  - shabble 12y ago
    > * To be honest, no. But Wikipedia seems to have that event pretty well covered* And in a lot of cases, the actual citations or references used by Wikipedia are dead links, or have some sort of 'last retrieved at' annotation, or end up at parked/squatted domains and the original content is no longer there to validate the claims it supposedly made. Offhand I don't know if Wikipedia has a policy on references to Internet Archive / wayback-machine links to things, but imo, being able to point to a snapshot of a reference at the time it was made, as well as the current version if it still exists, is a desirable feature, and well worth striving for. Internet pagerot/linkrot has caused me grief with bookmarks for things I didn't realise I needed again until years later when they were gone.
    - Mithrandir 12y ago
      > Offhand I don't know if Wikipedia has a policy on references to Internet Archive / wayback-machine links to things They do have a policy.[1] Additionally, IA has been specifically crawling links on Wikipedia to preserve the citations.[2][3] 1: https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine 2: https://blog.archive.org/2013/10/25/fixing-broken-links/ https://blog.archive.org/2013/10/25/fixing-broken-links/ 3: https://archive.org/details/NO404-WKP/v2 https://archive.org/details/NO404-WKP/v2

by moe 12y ago

ghuntley 12y ago

Well, I personally spent the better half of tonight recovering a bunch of old websites of mine from the IA using wget. IA is one of the most important initiatives/projects on the internet, links are not meant to change but they do and when it happens knowledge & history erodes. Remember the 2004 Indian Ocean earthquake and tsunami? It was the first major event that provided mainstream credibility to the blogosphere, before this disaster consumers turned to news publications for information. For the first time ever news publications did not have the info and were desperate for content, any. TV crews flew into the effected areas and started purchasing suitcases/video/still cameras sight unseen off shell shocked tourists whom were too traumatized to remember if they had captured anything. It was also the last major disaster that occurred before the social revolution - YouTube, Flickr, Facebook, smart-phones and mainstream adoption of broadband. Back then the main way for the general public to share information was via email (which had limits of 10mb) and if they were technical or knew someone technical on their 100mb of free hosting provided by their ISP. Now imagine you're a student, academic researcher, production assistant and you have been tasked researching what transpired. As of right now, the majority of content from 2004/the disaster is no-longer available on the internet and only accessible via the IA. (nb: I was the guy behind http://web.archive.org/web/20050711081524/http://www.waveofdestruction.org/ http://web.archive.org/web/20050711081524/http://www.waveofd... which was the first emergency crisis live/micro blog. For what Google now has a team of volunteers doing during a crisis I did it myself with a single pentium 4 cpu, 4GB of ram, a single IDE hard drive, a ramdrive, a 100mbit uplink and a week without sleep. WOD will be relaunching within the next couple of weeks and was one of the sites recovered from the IA.)

moe 12y ago
knowledge & history erodes There surely is history worth preserving. It's just this "compulsive hoarding"-approach that seems ultimately futile to me. The internet itself inherently preserves the knowledge that at least one person cares enough about to put it on a webserver. Pretty much everything of even minor relevance should be referenced by Wikipedia by now. IMHO the internet itself is the "Internet Archive". Remember the 2004 Indian Ocean earthquake and tsunami? To be honest, no. But Wikipedia seems to have that event pretty well covered; http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_and_tsunami http://en.wikipedia.org/wiki/2004_Indian_Ocean_earthquake_an...
- ghaff 12y ago
  There is some balance between "compulsive hoarding" and preservation. I actually agree that it's an ultimately futile effort--and not very fruitful--to obsess over preserving every bit of text, audio, and video that's ever been posted to the Internet. On the other hand, preserving the content of Usenet or even Geocities do strike me as worth some disk space and won't happen organically for the most part. And the sort of people who have the mindset to do that preservation probably inherently have a bit of the compulsive hoarder about them. On balance, I think it's a good thing.
- shabble 12y ago
  > * To be honest, no. But Wikipedia seems to have that event pretty well covered* And in a lot of cases, the actual citations or references used by Wikipedia are dead links, or have some sort of 'last retrieved at' annotation, or end up at parked/squatted domains and the original content is no longer there to validate the claims it supposedly made. Offhand I don't know if Wikipedia has a policy on references to Internet Archive / wayback-machine links to things, but imo, being able to point to a snapshot of a reference at the time it was made, as well as the current version if it still exists, is a desirable feature, and well worth striving for. Internet pagerot/linkrot has caused me grief with bookmarks for things I didn't realise I needed again until years later when they were gone.
  - Mithrandir 12y ago
    > Offhand I don't know if Wikipedia has a policy on references to Internet Archive / wayback-machine links to things They do have a policy.[1] Additionally, IA has been specifically crawling links on Wikipedia to preserve the citations.[2][3] 1: https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine https://en.wikipedia.org/wiki/Help:Using_the_Wayback_Machine 2: https://blog.archive.org/2013/10/25/fixing-broken-links/ https://blog.archive.org/2013/10/25/fixing-broken-links/ 3: https://archive.org/details/NO404-WKP/v2 https://archive.org/details/NO404-WKP/v2