6 ms·

Should the internet be archived? Seriously. It's a transient medium, somewhere between a beautifully illustrated book and a shopping list scrawled on the back

by ry_ry 9y ago

Should the internet be archived? Seriously. It's a transient medium, somewhere between a beautifully illustrated book and a shopping list scrawled on the back of a napkin in crayon, recited down a megaphone by a drunk. Let content live or die in its merit, rather than the desire to arbitrarily preserve it.

toomuchtodo 9y ago
Culture should be preserved. Let future generations judge its merits.
nxsynonym 9y ago
For as much great information that exists on the internet, there's 10x as much noise. I'm all for selective preservation - but I agree - no reason to preserve just for the sake of it. Not every book printed is worth reading, saving, and passing down. Not every thought is worth speaking and hearing. Not every meme, click bait article, or shitpost needs to be saved.
- eric_h 9y ago
  > there's 10x as much noise Honestly I think you're being extremely generous about how much signal there is in the noise. I think the decimal point needs to move at least two places to the right, but probably more like four.
- kalleboo 9y ago
  There's so much data that it's an impossible task to select what is worth preserving or not. For just the salary of one "preservation judge" you could just buy another couple thousand TB of storage and store it all
- kalleboo 9y ago
  > Not every meme, click bait article, or shitpost needs to be saved. Unless you want to unravel how Trump got elected?
WalterBright 9y ago
Just as a counterpoint, archeologists love to sift through the trash heaps of ancient civilizations.
- Pica_soO 9y ago
  I majored in 4chanarchology, i minored in Koprolith-studys.
  - noonespecial 9y ago
    The thought that there may one day be Pewdiepie scholars just as there are Milton scholars today fills me with a sort of fascinated dread.
  - lmm 9y ago
    I got contacted by some internet historians because I had the oldest extant archive of one of 4chan's erotica boards. That was a weird feeling.
- jameshart 9y ago
  I'm sure if they could find them they would prefer to sift through the carefully curated museum archives of ancient civilizations, but the trash heaps tend to be what we find.
  - pharrington 9y ago
    Hardly prefer; though the comparison of the ancient's curations with their general first-hard records and remains does provide fantastic insight as to the biases of the ancient archivers.
    - jfoutz 9y ago
      I thought Egypt was a good example of this. The monuments to power are easy to find, but day to day life of people, and their views and opinions about the world are much tougher to come by. The stuff people took the time to carve in stone survived. And not much else.
Veratyr 9y ago
The internet should be at least partially archived, the problem is determining which part to archive. 1000 years from now, what do we think will be of the most use to historians and our descendants? I think the big problem here is that the durability of data is primarily based on a corporation surviving. Think of all that will be lost if Twitter goes bankrupt. Plus the things that will be useful in the future might not seem useful now, so it may not seem worth preserving them.
pmoriarty 9y ago
"Should the internet be archived?" This would be a great question to ask researchers in archaeology, anthropology, linguistics, history, etc. My own impression is that the more data there is, the better for researchers. They can always weed out what they don't need later, but there's no way to get back information that has been destroyed or lost. Also, in the past people have made some horrible choices regarding what's valuable and what's not. For all sorts of reasons, from political to psychological to social, they've done things like burn, discard, or destroy texts and artwork we now consider valuable but they did not. Often, what was considered valuable (like sacred texts) was not really as revealing about their daily lives as things they did not value and discarded. So I'd really think twice before we discard any records. If there's some kind of serious pressing need to be selective (such as lack of space) that's one thing. But if keeping it more would not be a huge burden, I'm much more of an inclusionist than a deletionist.
- mindcrime 9y ago
  Also, in the past people have made some horrible choices regarding what's valuable and what's not. For all sorts of reasons, from political to psychological to social, they've done things like burn, discard, or destroy texts and artwork we now consider valuable but they did not. Case in point: https://en.wikipedia.org/wiki/Doctor_Who_missing_episodes https://en.wikipedia.org/wiki/Doctor_Who_missing_episodes
  - Consultant32452 9y ago
    Don't forget the original moon landing recordings. How could anyone have thought so little of those that we lost them?
    - tdburn 9y ago
      That's because we never really landed on the moon!
      - ant6n 9y ago
Springtime 9y ago
> Let content live or die in its merit, rather than the desire to arbitrarily preserve it. The number of times per week I need to find a link that's gone dead—even occasionally for fairly recent articles—demonstrates the practical need for preserving web content, even just for pure contextual reasons. There's an enormous amount of useful content archived: information, original documentation, downloads, culture. I have a lot of respect for internet archivists.
joe_the_user 9y ago
Of course the Internet should be archived. A large portion of the Internet will be archived regardless. Not archiving all of it just allows a more distorted picture of what happened. Any piece of information that's produced is not simply "content" having "merit" but evidence for the structure of this society as a whole. The most ridiculous content still sheds light on other content and gives an explanation for the origin of still other content and even events. And, of course, even if information has a set merit, it won't necessarily survive or die on that merit but rather on a combination of merit, who it money for, purely random factors and so-forth.
edgarvaldes 9y ago
> Should the internet be archived? Absolutely. Without a doubt. And not because the merit of a particular piece.
crygin 9y ago
Imagine our understanding of evolutionary biology if we had access to the entire corpus of failed mutations, the whole nondeterministic tree instead of the single surviving branch. This is that for the marketplace of ideas.
jccalhoun 9y ago
I've been working on a project on 90s gaming (mostly centered around the FPS genre). It is incredibly frustrating that so many of the sites from then are gone. I will find a reference to an interview or something and even archive.org doesn't have it. Or the surprising number of RealAudio proto-podcasts that archive.org doesn't have archived. Those would be incredibly useful to have. (I even went so far as to find the owner of one of the defunct sites and tweeted him about the files. He said they were in a storage unit somewhere and he would get around to reuploading them "eventually.")
- ChickeNES 9y ago
  I'm working on-and-off on something not totally unrelated and the number of times that a site has been archived, disappeared, but then replaced by a domain parked page with robots.txt restrictions is infuriating. The data is there, but with no way to access it. I understand obeying robots.txt when crawling, but applying it to existing content is just idiotic.
krapp 9y ago
>Let content live or die in its merit, rather than the desire to arbitrarily preserve it. But content in every other medium has lived and died due to happenstance, religion or the aesthetic tastes of the wealthy. Why should the internet be an exception?
codingdave 9y ago
But it is difficult to tell what has merit when looking in the short term. If you archive as much as you can, you can sift through it all later when history tells you what become important. Or it doesn't even have to be profound... I was talking to someone yesterday who mentioned that their corporate site had actually been online since the early 90s, despite them being a non-tech company, and he wished he could show us the old designs. So we pulled up some old designs on archive.org. They certainly weren't archived for their historical value, but it was a nice experience.
ernsheong 9y ago
(shameless plug) I am working on PageDash (https://www.pagedash.com https://www.pagedash.com, just a Google form for now). The idea is to archive web pages for your private use via one-click via a browser (Chrome first) extension. So I think it jives with that people are saying here: archive what you think is important to you, leave the rest. PageDash takes a different approach in that it doesn't use a browser in the cloud (usually PhantomJS) to archive the pages. This means that PageDash can archive when you are logged in, or when you are on a private network, for example. PageDash tries its best to preserve the page exactly as you saw it. Sign up to be notified when it launches!
- badsectoracula 9y ago
  How does it compare with Mozilla Archive Format (MAFF)? MAFF saves entire pages (or sets of pages, by saving all tabs) with all the resources, etc, in a single archive file. I often use it to save articles, tutorials, etc that have a ton of images.
  - ernsheong 9y ago
    Haven't used MAFF before. But PageDash will preserve the look and feel exactly, and handle more corner cases. Web is huge, perfection is almost impossible, but we'll get there. Plans are to also allow MHTML/PDF/JPG exports one day. More importantly PageDash is web hosted and will allow users to share their archived links (private by default). Eventually, full-text search and shared folders can enable collaboration. Basically stuff that an offline-only solution can't do.
- voltagex_ 9y ago
  What's your plan for when (not if) PageDash shuts down? what happens to the data?
  - ernsheong 9y ago
    Well, we'll have to start somewhere. At the end of the day, someone has to be a guardian, even if you're the Internet Archive. Ideally, PageDash turns itself over to be a non-profit sort of thing. Else, a corporation takes over. If really shutting down, there will be an export option for users for a graceful shutdown.
noufalibrahim 9y ago
As someone who's worked on the wayback machine and responsible for uploading around 300 TB of data into it, I think it should be. From all the well researched, immensely valuable sites and page to all the badly photoshopped memes and ugliness that simmers in it. I like to think of as the collective external brain of humanity. The value depends on the people who are trying to dig up information about their ancestors to find out how they managed to live in those primitive times. Historians could always use more information from multiple sources to piece together accurate pictures of times past and the reasons why people did what they did. We know that since we have dedicated people trying to reconstruct our pasts. We'll have to live with what survived from what our ancestors left us but there's no reason to limit generations of future scientists to such scant and badly pieced together material.

by ry_ry 9y ago

toomuchtodo 9y ago

Culture should be preserved. Let future generations judge its merits.

nxsynonym 9y ago

For as much great information that exists on the internet, there's 10x as much noise. I'm all for selective preservation - but I agree - no reason to preserve just for the sake of it. Not every book printed is worth reading, saving, and passing down. Not every thought is worth speaking and hearing. Not every meme, click bait article, or shitpost needs to be saved.

eric_h 9y ago
> there's 10x as much noise Honestly I think you're being extremely generous about how much signal there is in the noise. I think the decimal point needs to move at least two places to the right, but probably more like four.
kalleboo 9y ago
There's so much data that it's an impossible task to select what is worth preserving or not. For just the salary of one "preservation judge" you could just buy another couple thousand TB of storage and store it all
kalleboo 9y ago
> Not every meme, click bait article, or shitpost needs to be saved. Unless you want to unravel how Trump got elected?

WalterBright 9y ago

Just as a counterpoint, archeologists love to sift through the trash heaps of ancient civilizations.

Pica_soO 9y ago
I majored in 4chanarchology, i minored in Koprolith-studys.
- noonespecial 9y ago
  The thought that there may one day be Pewdiepie scholars just as there are Milton scholars today fills me with a sort of fascinated dread.
- lmm 9y ago
  I got contacted by some internet historians because I had the oldest extant archive of one of 4chan's erotica boards. That was a weird feeling.
jameshart 9y ago
I'm sure if they could find them they would prefer to sift through the carefully curated museum archives of ancient civilizations, but the trash heaps tend to be what we find.
- pharrington 9y ago
  Hardly prefer; though the comparison of the ancient's curations with their general first-hard records and remains does provide fantastic insight as to the biases of the ancient archivers.
  - jfoutz 9y ago
    I thought Egypt was a good example of this. The monuments to power are easy to find, but day to day life of people, and their views and opinions about the world are much tougher to come by. The stuff people took the time to carve in stone survived. And not much else.

Veratyr 9y ago

The internet should be at least partially archived, the problem is determining which part to archive. 1000 years from now, what do we think will be of the most use to historians and our descendants? I think the big problem here is that the durability of data is primarily based on a corporation surviving. Think of all that will be lost if Twitter goes bankrupt. Plus the things that will be useful in the future might not seem useful now, so it may not seem worth preserving them.

pmoriarty 9y ago

"Should the internet be archived?" This would be a great question to ask researchers in archaeology, anthropology, linguistics, history, etc. My own impression is that the more data there is, the better for researchers. They can always weed out what they don't need later, but there's no way to get back information that has been destroyed or lost. Also, in the past people have made some horrible choices regarding what's valuable and what's not. For all sorts of reasons, from political to psychological to social, they've done things like burn, discard, or destroy texts and artwork we now consider valuable but they did not. Often, what was considered valuable (like sacred texts) was not really as revealing about their daily lives as things they did not value and discarded. So I'd really think twice before we discard any records. If there's some kind of serious pressing need to be selective (such as lack of space) that's one thing. But if keeping it more would not be a huge burden, I'm much more of an inclusionist than a deletionist.

mindcrime 9y ago
Also, in the past people have made some horrible choices regarding what's valuable and what's not. For all sorts of reasons, from political to psychological to social, they've done things like burn, discard, or destroy texts and artwork we now consider valuable but they did not. Case in point: https://en.wikipedia.org/wiki/Doctor_Who_missing_episodes https://en.wikipedia.org/wiki/Doctor_Who_missing_episodes
- Consultant32452 9y ago
  Don't forget the original moon landing recordings. How could anyone have thought so little of those that we lost them?
  - tdburn 9y ago
    That's because we never really landed on the moon!
    - ant6n 9y ago

Springtime 9y ago

> Let content live or die in its merit, rather than the desire to arbitrarily preserve it. The number of times per week I need to find a link that's gone dead—even occasionally for fairly recent articles—demonstrates the practical need for preserving web content, even just for pure contextual reasons. There's an enormous amount of useful content archived: information, original documentation, downloads, culture. I have a lot of respect for internet archivists.

joe_the_user 9y ago

Of course the Internet should be archived. A large portion of the Internet will be archived regardless. Not archiving all of it just allows a more distorted picture of what happened. Any piece of information that's produced is not simply "content" having "merit" but evidence for the structure of this society as a whole. The most ridiculous content still sheds light on other content and gives an explanation for the origin of still other content and even events. And, of course, even if information has a set merit, it won't necessarily survive or die on that merit but rather on a combination of merit, who it money for, purely random factors and so-forth.

edgarvaldes 9y ago

> Should the internet be archived? Absolutely. Without a doubt. And not because the merit of a particular piece.

crygin 9y ago

Imagine our understanding of evolutionary biology if we had access to the entire corpus of failed mutations, the whole nondeterministic tree instead of the single surviving branch. This is that for the marketplace of ideas.

jccalhoun 9y ago

I've been working on a project on 90s gaming (mostly centered around the FPS genre). It is incredibly frustrating that so many of the sites from then are gone. I will find a reference to an interview or something and even archive.org doesn't have it. Or the surprising number of RealAudio proto-podcasts that archive.org doesn't have archived. Those would be incredibly useful to have. (I even went so far as to find the owner of one of the defunct sites and tweeted him about the files. He said they were in a storage unit somewhere and he would get around to reuploading them "eventually.")

ChickeNES 9y ago
I'm working on-and-off on something not totally unrelated and the number of times that a site has been archived, disappeared, but then replaced by a domain parked page with robots.txt restrictions is infuriating. The data is there, but with no way to access it. I understand obeying robots.txt when crawling, but applying it to existing content is just idiotic.

krapp 9y ago

>Let content live or die in its merit, rather than the desire to arbitrarily preserve it. But content in every other medium has lived and died due to happenstance, religion or the aesthetic tastes of the wealthy. Why should the internet be an exception?

codingdave 9y ago

But it is difficult to tell what has merit when looking in the short term. If you archive as much as you can, you can sift through it all later when history tells you what become important. Or it doesn't even have to be profound... I was talking to someone yesterday who mentioned that their corporate site had actually been online since the early 90s, despite them being a non-tech company, and he wished he could show us the old designs. So we pulled up some old designs on archive.org. They certainly weren't archived for their historical value, but it was a nice experience.

ernsheong 9y ago

(shameless plug) I am working on PageDash (https://www.pagedash.com https://www.pagedash.com, just a Google form for now). The idea is to archive web pages for your private use via one-click via a browser (Chrome first) extension. So I think it jives with that people are saying here: archive what you think is important to you, leave the rest. PageDash takes a different approach in that it doesn't use a browser in the cloud (usually PhantomJS) to archive the pages. This means that PageDash can archive when you are logged in, or when you are on a private network, for example. PageDash tries its best to preserve the page exactly as you saw it. Sign up to be notified when it launches!

badsectoracula 9y ago
How does it compare with Mozilla Archive Format (MAFF)? MAFF saves entire pages (or sets of pages, by saving all tabs) with all the resources, etc, in a single archive file. I often use it to save articles, tutorials, etc that have a ton of images.
- ernsheong 9y ago
  Haven't used MAFF before. But PageDash will preserve the look and feel exactly, and handle more corner cases. Web is huge, perfection is almost impossible, but we'll get there. Plans are to also allow MHTML/PDF/JPG exports one day. More importantly PageDash is web hosted and will allow users to share their archived links (private by default). Eventually, full-text search and shared folders can enable collaboration. Basically stuff that an offline-only solution can't do.
voltagex_ 9y ago
What's your plan for when (not if) PageDash shuts down? what happens to the data?
- ernsheong 9y ago
  Well, we'll have to start somewhere. At the end of the day, someone has to be a guardian, even if you're the Internet Archive. Ideally, PageDash turns itself over to be a non-profit sort of thing. Else, a corporation takes over. If really shutting down, there will be an export option for users for a graceful shutdown.

noufalibrahim 9y ago

As someone who's worked on the wayback machine and responsible for uploading around 300 TB of data into it, I think it should be. From all the well researched, immensely valuable sites and page to all the badly photoshopped memes and ugliness that simmers in it. I like to think of as the collective external brain of humanity. The value depends on the people who are trying to dig up information about their ancestors to find out how they managed to live in those primitive times. Historians could always use more information from multiple sources to piece together accurate pictures of times past and the reasons why people did what they did. We know that since we have dedicated people trying to reconstruct our pasts. We'll have to live with what survived from what our ancestors left us but there's no reason to limit generations of future scientists to such scant and badly pieced together material.