4 ms·

I feel like it's the exact opposite. We're drowning in data. Consider: the Library of Congress has (literally) a copy of every work that has ever been in print

I feel like it's the exact opposite. We're drowning in data. Consider: the Library of Congress has (literally) a copy of every work that has ever been in print in the U.S. The Internet Archive has archived much of the public internet. There are other resources like arXiv, Zenodo, etc. Many of these organizations are very conscientious about data retention and longevity. Is there even more data that could be saved in principle? Of course. But the amount of data we're talking about is already so much dramatically larger than what we have about, say, the antiquities, that it seems like an absurd comparison. The problem in the future (excluding extinction-level events) will be wading through that data, which even in the handful of resources I've mentioned above will be so far beyond any single human's ability to understand that it's just unfathomable.

gunapologist99 3y ago
A single EMP event, like the Carrington Event, or just a single launch of something like Starfish Prime, could wipe all of that out in the blink of an eye. Something like this occurring seems inevitable given a long enough timeline, and 500 years doesn't seem that far away.
tivert 3y ago
>> With fewer and fewer printed works, I wonder how future archeologists will discover human knowledge from our era. Especially after a world-wide catastrophe. Human language written in the physical world seems more durable long-term than the electrons in a flash drive. > Consider: the Library of Congress has (literally) a copy of every work that has ever been in print in the U.S. The Internet Archive has archived much of the public internet. Those are both very centralized. The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. > Is there even more data that could be saved in principle? Of course. But the amount of data we're talking about is already so much dramatically larger than what we have about, say, the antiquities, that it seems like an absurd comparison. That sheer amount of data may work against preservation in another way: too much noise distracting from identifying and preserving the valuable stuff. > The problem in the future (excluding extinction-level events) will be wading through that data, which even in the handful of resources I've mentioned above will be so far beyond any single human's ability to understand that it's just unfathomable. Not necessarily: all that data is very fragile. Besides format conversion and migration issues (about that flash-based website...), you have site/platform longevity, and the time-bomb of cloud hosting costs (someone stops paying a bill? Poof, it's gone). Then you have the spectre of a dark age: a few decades of a teetering industrial base after a major world war that destroys a couple of high-tech manufacturing centers, and that data is pretty much all gone.
- Intralexical 3y ago
  > The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. MDisc definitely can, if its claims are to be believed. Maybe you can find some specialized tape too? Hard disks and normal optical disks probably could too, as long as you have a sufficiently large redundant array of them with error-detection on your bookshelf— If it can't last that long, just keep doubling the number of starting drives until it can. If you want to get fancy, you can try to pay for something around HD-Rosetta, or Microsoft/Hitachi/Soton's printed glass tech. And then of course if you wanna get simple, you could always stick your digital data into a mountable book/scroll of printed QR codes (or equivalent). Of course, none of these options are actually deployed at any significant scale. But I'm not sure you can realistically expect any arbitrary book to last for centuries either. IIRC Modern print media uses acidic paper as a cost-cutting measure, causing it to become brittle and crumble over time… https://en.wikipedia.org/wiki/Acidic_paper https://en.wikipedia.org/wiki/Acidic_paper "[…] causes huge losses in library and archives collection […] 90% of the resources published by the mid-1990s […] have all the features of acidic paper. […] established to care for the heritage of the past, are not able to effectively carry out their mission […] it is not possible to save all the documents from the 19th and 20th centuries […] In recent years, most books have been printed on acid-free paper […]".
  - tivert 3y ago
    >> The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. > MDisc definitely can, if its claims are to be believed. Maybe you can find some specialized tape too? 1) I don't think they actually make real M-Discs anymore (the DVDs). There are some M-Disc branded BD-Rs, but I don't think those use any different technology than standard BD-Rs. 2) They're not stand-alone, and by that I mean they need a player or they're unreadable. IMHO, it's unlikely there will be many working DVD/Blu-ray players in 100 years, and very unlikely there will be any players for "some specialized tape." I've been working a bit with VHS recently, and it's eye opening to see a once ubiquitous format start to drift into unreadability (it's not there yet, but a lot of the specialized equipment needed to do a good job is becoming expensive, hard-to-find, and breaking down).
jacquesm 3y ago
We're drowning in data in the present. But I predict that five hundred years from now these will be known as 'the lost years of history' because there wasn't enough long term durable storage dedicated to preserving events that we do not think worth preserving today. But you'll have a million AI generated recipes for Apple Pie from spam websites of the era to console you.
- rbanffy 3y ago
  > But you'll have a million AI generated recipes for Apple Pie from spam websites of the era to console you. And good luck finding the relevant information in the middle of all that noise.
  - titzer 3y ago
    This is exactly the problem; information isn't necessarily being lost, but silted over and sometimes intentionally buried. Part of it is due to its natural loss of relevance, part of it is due to loss of popularity and attention, and part of it is deliberate commercial motive.
    - rbanffy 3y ago
      > information isn't necessarily being lost, but silted over and sometimes intentionally buried. And that kind of touches disinformation campaigns. There is a lot of noise being deliberately and maliciously added so that it out represents any information someone wants to suppress. An AI model trained on this corpus will have all the wrong ideas.
    - bdw5204 3y ago
      Information is being lost constantly on the internet. Whether it is CNET deleting old articles as an SEO tactic[0], domain names expiring, formerly popular sites like Geocities being erased altogether, Google mismanaging its Usenet archive or once popular blogs getting deleted for TOS or account inactivity issues, the internet is certainly not forever. Archive.org isn't really a solution either because it is not uncommon for domain squatters to use a robots.txt setting to get them to remove the domain from the Wayback Machine. You can't even rely on large social media platforms because people delete their accounts, some people auto-delete their old social media posts and platforms decide to login-wall themselves like what happened with Twitter. Link rot is a major problem that people don't recognize, especially for information that was only ever online. Most of the obscure web sites I used to read and hang out on are gone and many of the things I remember are now completely unverifiable because I didn't save a copy of every web site that ever influenced me. My own unfinished game project from my teenage years vanished from the internet without a trace after I lost interest and I lost all of the code along with all of my other data from my teen years in a hard drive crash around the time I finished high school. My mods I made for games and never distributed are sitting on old laptops in my closet I haven't even turned on in years that may or may not even work anymore. I imagine everybody else who's been heavily online in the past has similar stories of just how ephemeral digital information is. [0]: https://www.theverge.com/2023/8/9/23826342/cnet-content-pruning-deleting-articles-google-seo https://www.theverge.com/2023/8/9/23826342/cnet-content-prun...
nocman 3y ago
> Consider: the Library of Congress has (literally) a copy of every work that has ever been in print in the U.S. This is not true. The Library of Congress has a copy of works that have been registered for copyright with them. I am sure millions of works are created in the US every year that never get registered with the Library of Congress. While a registration with the LOC isn't a bad idea, in the USA at least authors of written works automatically have copyright of their work, regardless of whether they have registered it for copyright with the LOC.

gunapologist99 3y ago

A single EMP event, like the Carrington Event, or just a single launch of something like Starfish Prime, could wipe all of that out in the blink of an eye. Something like this occurring seems inevitable given a long enough timeline, and 500 years doesn't seem that far away.

tivert 3y ago

>> With fewer and fewer printed works, I wonder how future archeologists will discover human knowledge from our era. Especially after a world-wide catastrophe. Human language written in the physical world seems more durable long-term than the electrons in a flash drive. > Consider: the Library of Congress has (literally) a copy of every work that has ever been in print in the U.S. The Internet Archive has archived much of the public internet. Those are both very centralized. The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. > Is there even more data that could be saved in principle? Of course. But the amount of data we're talking about is already so much dramatically larger than what we have about, say, the antiquities, that it seems like an absurd comparison. That sheer amount of data may work against preservation in another way: too much noise distracting from identifying and preserving the valuable stuff. > The problem in the future (excluding extinction-level events) will be wading through that data, which even in the handful of resources I've mentioned above will be so far beyond any single human's ability to understand that it's just unfathomable. Not necessarily: all that data is very fragile. Besides format conversion and migration issues (about that flash-based website...), you have site/platform longevity, and the time-bomb of cloud hosting costs (someone stops paying a bill? Poof, it's gone). Then you have the spectre of a dark age: a few decades of a teetering industrial base after a major world war that destroys a couple of high-tech manufacturing centers, and that data is pretty much all gone.

Intralexical 3y ago
> The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. MDisc definitely can, if its claims are to be believed. Maybe you can find some specialized tape too? Hard disks and normal optical disks probably could too, as long as you have a sufficiently large redundant array of them with error-detection on your bookshelf— If it can't last that long, just keep doubling the number of starting drives until it can. If you want to get fancy, you can try to pay for something around HD-Rosetta, or Microsoft/Hitachi/Soton's printed glass tech. And then of course if you wanna get simple, you could always stick your digital data into a mountable book/scroll of printed QR codes (or equivalent). Of course, none of these options are actually deployed at any significant scale. But I'm not sure you can realistically expect any arbitrary book to last for centuries either. IIRC Modern print media uses acidic paper as a cost-cutting measure, causing it to become brittle and crumble over time… https://en.wikipedia.org/wiki/Acidic_paper https://en.wikipedia.org/wiki/Acidic_paper "[…] causes huge losses in library and archives collection […] 90% of the resources published by the mid-1990s […] have all the features of acidic paper. […] established to care for the heritage of the past, are not able to effectively carry out their mission […] it is not possible to save all the documents from the 19th and 20th centuries […] In recent years, most books have been printed on acid-free paper […]".
- tivert 3y ago
  >> The benefit of print media is its forced-decentralization and stand-alone nature. It can survive for 100+ years on a shelf, and I don't think any digital data format could match that. > MDisc definitely can, if its claims are to be believed. Maybe you can find some specialized tape too? 1) I don't think they actually make real M-Discs anymore (the DVDs). There are some M-Disc branded BD-Rs, but I don't think those use any different technology than standard BD-Rs. 2) They're not stand-alone, and by that I mean they need a player or they're unreadable. IMHO, it's unlikely there will be many working DVD/Blu-ray players in 100 years, and very unlikely there will be any players for "some specialized tape." I've been working a bit with VHS recently, and it's eye opening to see a once ubiquitous format start to drift into unreadability (it's not there yet, but a lot of the specialized equipment needed to do a good job is becoming expensive, hard-to-find, and breaking down).

jacquesm 3y ago

We're drowning in data in the present. But I predict that five hundred years from now these will be known as 'the lost years of history' because there wasn't enough long term durable storage dedicated to preserving events that we do not think worth preserving today. But you'll have a million AI generated recipes for Apple Pie from spam websites of the era to console you.

rbanffy 3y ago
> But you'll have a million AI generated recipes for Apple Pie from spam websites of the era to console you. And good luck finding the relevant information in the middle of all that noise.
- titzer 3y ago
  This is exactly the problem; information isn't necessarily being lost, but silted over and sometimes intentionally buried. Part of it is due to its natural loss of relevance, part of it is due to loss of popularity and attention, and part of it is deliberate commercial motive.
  - rbanffy 3y ago
    > information isn't necessarily being lost, but silted over and sometimes intentionally buried. And that kind of touches disinformation campaigns. There is a lot of noise being deliberately and maliciously added so that it out represents any information someone wants to suppress. An AI model trained on this corpus will have all the wrong ideas.
  - bdw5204 3y ago
    Information is being lost constantly on the internet. Whether it is CNET deleting old articles as an SEO tactic[0], domain names expiring, formerly popular sites like Geocities being erased altogether, Google mismanaging its Usenet archive or once popular blogs getting deleted for TOS or account inactivity issues, the internet is certainly not forever. Archive.org isn't really a solution either because it is not uncommon for domain squatters to use a robots.txt setting to get them to remove the domain from the Wayback Machine. You can't even rely on large social media platforms because people delete their accounts, some people auto-delete their old social media posts and platforms decide to login-wall themselves like what happened with Twitter. Link rot is a major problem that people don't recognize, especially for information that was only ever online. Most of the obscure web sites I used to read and hang out on are gone and many of the things I remember are now completely unverifiable because I didn't save a copy of every web site that ever influenced me. My own unfinished game project from my teenage years vanished from the internet without a trace after I lost interest and I lost all of the code along with all of my other data from my teen years in a hard drive crash around the time I finished high school. My mods I made for games and never distributed are sitting on old laptops in my closet I haven't even turned on in years that may or may not even work anymore. I imagine everybody else who's been heavily online in the past has similar stories of just how ephemeral digital information is. [0]: https://www.theverge.com/2023/8/9/23826342/cnet-content-pruning-deleting-articles-google-seo https://www.theverge.com/2023/8/9/23826342/cnet-content-prun...

nocman 3y ago

> Consider: the Library of Congress has (literally) a copy of every work that has ever been in print in the U.S. This is not true. The Library of Congress has a copy of works that have been registered for copyright with them. I am sure millions of works are created in the US every year that never get registered with the Library of Congress. While a registration with the LOC isn't a bad idea, in the USA at least authors of written works automatically have copyright of their work, regardless of whether they have registered it for copyright with the LOC.